[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:56 dist-test-slave-g8zg krb5kdc[22987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:56 dist-test-slave-g8zg krb5kdc[22987](info): set up 2 sockets
Jun 19 06:52:56 dist-test-slave-g8zg krb5kdc[22987](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:52:57.635864 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:52:57 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:52:57.783021 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41585
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:41585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:57.906418 23000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:57.906955 23000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:57.907056 23000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:57.911355 23000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:57.911448 23000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:57.911487 23000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:57.911541 23000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:57.911577 23000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:57.917119 23000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:41585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:57.918462 23000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:57.919725 23000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:57.929993 23005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:57.930124 23000 server_base.cc:1034] running on GCE node
W20260619 06:52:57.930058 23008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:57.929994 23006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:57.931212 23000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:57.932528 23000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:57.933746 23000 hybrid_clock.cc:648] HybridClock initialized: now 1781851977933724 us; error 41 us; skew 500 ppm
Jun 19 06:52:57 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851977, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:57.936986 23000 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:52:57.937985 23000 webserver.cc:460] Webserver started at http://127.22.113.62:44137/ using document root <none> and password file <none>
I20260619 06:52:57.938614 23000 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:57.938709 23000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:57.938941 23000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:57.941269 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140"
format_stamp: "Formatted at 2026-06-19 06:52:57 on dist-test-slave-g8zg"
server_key: "e03bbaee2bf02d8f827b1c009f5ddf59"
server_key_iv: "9d78805d4b6142576cc08571b4875c7c"
server_key_version: "encryptionkey@0"
I20260619 06:52:57.941888 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140"
format_stamp: "Formatted at 2026-06-19 06:52:57 on dist-test-slave-g8zg"
server_key: "e03bbaee2bf02d8f827b1c009f5ddf59"
server_key_iv: "9d78805d4b6142576cc08571b4875c7c"
server_key_version: "encryptionkey@0"
I20260619 06:52:57.946128 23000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:52:57.949023 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:57.950601 23000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:52:57.950759 23000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140"
format_stamp: "Formatted at 2026-06-19 06:52:57 on dist-test-slave-g8zg"
server_key: "e03bbaee2bf02d8f827b1c009f5ddf59"
server_key_iv: "9d78805d4b6142576cc08571b4875c7c"
server_key_version: "encryptionkey@0"
I20260619 06:52:57.950879 23000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:57.965776 23000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:57.969338 23000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:57.969559 23000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:57.978034 23000 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:41585
I20260619 06:52:57.978153 23067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:41585 every 8 connection(s)
I20260619 06:52:57.979239 23000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:57.980948 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23000
I20260619 06:52:57.981145 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:52:57.982549 23068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:57.982643 22980 external_mini_cluster.cc:1428] Setting key ca1190c401da07a5a851362ab577f573
I20260619 06:52:57.989399 23068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: Bootstrap starting.
I20260619 06:52:57.991748 23068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:57.992528 23068 log.cc:824] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:57.994519 23068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: No bootstrap required, opened a new log
I20260619 06:52:57.998260 23068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } }
I20260619 06:52:57.998541 23068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:57.998626 23068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884ac2fc87ee4bd0aa8d6c87ddc4c140, State: Initialized, Role: FOLLOWER
I20260619 06:52:57.999122 23068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } }
I20260619 06:52:57.999306 23068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:57.999382 23068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:57.999490 23068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:58.000649 23068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } }
I20260619 06:52:58.001034 23068 leader_election.cc:302] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884ac2fc87ee4bd0aa8d6c87ddc4c140; no voters:
I20260619 06:52:58.001399 23068 leader_election.cc:288] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:58.001473 23073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:58.001673 23073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [term 1 LEADER]: Becoming Leader. State: Replica: 884ac2fc87ee4bd0aa8d6c87ddc4c140, State: Running, Role: LEADER
I20260619 06:52:58.002018 23073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } }
Jun 19 06:52:57 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:52:58.002775 23068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:58.003995 23074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } } }
I20260619 06:52:58.004283 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884ac2fc87ee4bd0aa8d6c87ddc4c140. Latest consensus state: current_term: 1 leader_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884ac2fc87ee4bd0aa8d6c87ddc4c140" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41585 } } }
I20260619 06:52:58.004443 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:58.004367 23074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140 [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:58.010076 23088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:58.010169 23088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:58.010244 23086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:58.010259 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:57.986520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.22.113.62:41585)
0619 06:52:57.987046 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:57.987064 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:57.988436 (+ 1372us) server_negotiation.cc:407] Connection header received
0619 06:52:57.992158 (+ 3722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:57.992176 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:57.992518 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:57.992891 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:57.995307 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:57.996392 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:57.997623 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:57.998120 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:57.998312 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.003303 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.003313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.003330 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.003368 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.005849 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.006851 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.006862 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.006871 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.007058 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.007517 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.007521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.007523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.007975 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.008208 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.008841 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.009172 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":157,"threads_started":1}
I20260619 06:52:58.011222 23086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:58.017915 23086 catalog_manager.cc:1347] Generated new cluster ID: 92c39099cb234918b6fc129f47390c0b
I20260619 06:52:58.018008 23086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:58.030709 23086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:58.031783 23086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:58.039707 23086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884ac2fc87ee4bd0aa8d6c87ddc4c140: Generated new TSK 0
I20260619 06:52:58.040442 23086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:52:58.079114 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:58.218093 23094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:58.218385 23094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:58.218432 23094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:58.222453 23094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:58.222518 23094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:58.222579 23094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:52:58.228080 23094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:58.229171 23094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:58.230191 23094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:58.241362 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:58.241353 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:58.241355 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:58.241757 23094 server_base.cc:1034] running on GCE node
I20260619 06:52:58.242206 23094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:58.242936 23094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:58.244132 23094 hybrid_clock.cc:648] HybridClock initialized: now 1781851978244111 us; error 38 us; skew 500 ppm
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:58.247622 23094 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:52:58.248730 23094 webserver.cc:460] Webserver started at http://127.22.113.1:43105/ using document root <none> and password file <none>
I20260619 06:52:58.249387 23094 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:58.249451 23094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:58.249663 23094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:58.252120 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "8fcde76e086f44d5bf8702260e83fd3a"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "374176141aae70465ff07ad7f4279b32"
server_key_iv: "73ce81678a5ec643b3d5bfc816bfbc1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.252702 23094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "8fcde76e086f44d5bf8702260e83fd3a"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "374176141aae70465ff07ad7f4279b32"
server_key_iv: "73ce81678a5ec643b3d5bfc816bfbc1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.257242 23094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:52:58.260082 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.261591 23094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:58.261703 23094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "8fcde76e086f44d5bf8702260e83fd3a"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "374176141aae70465ff07ad7f4279b32"
server_key_iv: "73ce81678a5ec643b3d5bfc816bfbc1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.261801 23094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:58.274767 23094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:58.278384 23094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:58.278576 23094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:58.279206 23094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:58.280146 23094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:58.280195 23094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.280241 23094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:58.280256 23094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.290799 23094 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:42385
I20260619 06:52:58.290889 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:42385 every 8 connection(s)
I20260619 06:52:58.292006 23094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:58.298350 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23094
I20260619 06:52:58.298517 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:58.298851 22980 external_mini_cluster.cc:1428] Setting key 1d6b5c3e30845a6c75da50fdde0db118
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:52:58.309283 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.294169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:46547 (local address 127.22.113.62:41585)
0619 06:52:58.294314 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:58.294319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:58.295466 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 06:52:58.297002 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:58.297006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:58.297065 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:58.297153 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:58.298524 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.299305 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.300882 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.301152 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.301291 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.304712 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.304717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.304720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.304755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.306512 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.307377 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.307381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.307383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.307472 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.307921 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.307924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.307925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.308093 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.308174 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.308992 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.309112 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:52:58.310699 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.294607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41585 (local address 127.22.113.1:46547)
0619 06:52:58.295208 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:58.295330 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:58.296768 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:58.297346 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:58.297354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:58.297801 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:58.298365 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.298379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.299446 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.299450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.300743 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.300753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.301338 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.301343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.301524 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.302206 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:58.302224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:58.304463 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:58.306662 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.306673 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.306696 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.307162 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.307598 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.307604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.307609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.307804 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.308212 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:58.308222 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:58.308653 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 06:52:58.308928 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 06:52:58.309362 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":140,"threads_started":1}
I20260619 06:52:58.312207 23223 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41585
I20260619 06:52:58.312554 23223 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:52:58.313182 23223 heartbeater.cc:511] Master 127.22.113.62:41585 requested a full tablet report, sending...
I20260619 06:52:58.314859 23032 ts_manager.cc:194] Registered new tserver with Master: 8fcde76e086f44d5bf8702260e83fd3a (127.22.113.1:42385)
I20260619 06:52:58.316499 23032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:46547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:52:58.331525 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:58.459645 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:58.460003 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:58.460084 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:58.464358 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:58.464463 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:58.464564 23228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:52:58.470480 23228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:58.471895 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:58.473181 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:58.485172 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:58.485157 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:58.485224 23228 server_base.cc:1034] running on GCE node
W20260619 06:52:58.485163 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:58.485961 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:58.486742 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:58.487917 23228 hybrid_clock.cc:648] HybridClock initialized: now 1781851978487834 us; error 105 us; skew 500 ppm
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:58.491334 23228 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:52:58.492383 23228 webserver.cc:460] Webserver started at http://127.22.113.2:42415/ using document root <none> and password file <none>
I20260619 06:52:58.493106 23228 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:58.493204 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:58.493489 23228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:58.495970 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "63a34ea04d1c4d4e96ad2144a586b878"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "edb08b4e864a5ab17bf6650417954f2f"
server_key_iv: "ed884fd8ad7ad7bbdc25f3ece9580133"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.496635 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "63a34ea04d1c4d4e96ad2144a586b878"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "edb08b4e864a5ab17bf6650417954f2f"
server_key_iv: "ed884fd8ad7ad7bbdc25f3ece9580133"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.501204 23228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:52:58.504400 23243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.506047 23228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:58.506192 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "63a34ea04d1c4d4e96ad2144a586b878"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "edb08b4e864a5ab17bf6650417954f2f"
server_key_iv: "ed884fd8ad7ad7bbdc25f3ece9580133"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.506345 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:58.523501 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:58.526680 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:58.526849 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:58.527448 23228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:58.528281 23228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:58.528331 23228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.528371 23228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:58.528398 23228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.538514 23228 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41259
I20260619 06:52:58.538617 23356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41259 every 8 connection(s)
I20260619 06:52:58.539741 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:58.542223 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23228
I20260619 06:52:58.542340 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:58.542615 22980 external_mini_cluster.cc:1428] Setting key c79aa164ac60709b51dc4f2e3dbf6505
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:52:58.557013 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.542027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:50791 (local address 127.22.113.62:41585)
0619 06:52:58.542229 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:58.542235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:58.543307 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 06:52:58.544972 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:58.544975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:58.545026 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:58.545106 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:58.546664 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.547199 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.549138 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.549459 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.549555 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.552837 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.552840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.552842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.552865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.554472 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.555140 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.555143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.555145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.555306 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.555675 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.555679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.555680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.555876 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.555959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.556652 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.556813 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:52:58.558194 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.542525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41585 (local address 127.22.113.2:50791)
0619 06:52:58.543082 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:58.543140 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:58.544750 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:58.545296 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:58.545303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:58.545741 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:58.546484 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.546505 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.547457 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.547464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.548998 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.549012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.549644 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.549649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.549869 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.550595 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:58.550625 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:58.552638 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:58.554614 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.554619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.554633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.555030 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.555404 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.555407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.555408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.555551 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.555989 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:58.555995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:58.556331 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 06:52:58.556569 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 06:52:58.556961 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":181,"threads_started":1}
I20260619 06:52:58.559626 23357 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41585
I20260619 06:52:58.559983 23357 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:58.560705 23357 heartbeater.cc:511] Master 127.22.113.62:41585 requested a full tablet report, sending...
I20260619 06:52:58.562084 23032 ts_manager.cc:194] Registered new tserver with Master: 63a34ea04d1c4d4e96ad2144a586b878 (127.22.113.2:41259)
I20260619 06:52:58.562734 23032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:50791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:52:58.575398 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:58.697899 23362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:58.698278 23362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:58.698362 23362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:58.702652 23362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:58.702741 23362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:58.702843 23362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:52:58.708678 23362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:58.710052 23362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:58.711320 23362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:58.722383 23370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:58.722395 23368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:58.723122 23362 server_base.cc:1034] running on GCE node
W20260619 06:52:58.722383 23367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:58.723748 23362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:58.724577 23362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:58.725780 23362 hybrid_clock.cc:648] HybridClock initialized: now 1781851978725757 us; error 47 us; skew 500 ppm
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:58.728945 23362 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:52:58.729929 23362 webserver.cc:460] Webserver started at http://127.22.113.3:43079/ using document root <none> and password file <none>
I20260619 06:52:58.730567 23362 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:58.730657 23362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:58.730921 23362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:58.733237 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "48647e45b9d2432890dc30839fa517f0"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "bab172c48617402c90b54e0e6a6394f9"
server_key_iv: "e99e644312c830b8f01bd17e58251973"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.733851 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "48647e45b9d2432890dc30839fa517f0"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "bab172c48617402c90b54e0e6a6394f9"
server_key_iv: "e99e644312c830b8f01bd17e58251973"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.738224 23362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:52:58.741235 23377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.742848 23362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:52:58.743010 23362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "48647e45b9d2432890dc30839fa517f0"
format_stamp: "Formatted at 2026-06-19 06:52:58 on dist-test-slave-g8zg"
server_key: "bab172c48617402c90b54e0e6a6394f9"
server_key_iv: "e99e644312c830b8f01bd17e58251973"
server_key_version: "encryptionkey@0"
I20260619 06:52:58.743136 23362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:58.766634 23362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:58.770155 23362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:58.770385 23362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:58.771036 23362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:58.772012 23362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:58.772091 23362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.772168 23362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:58.772217 23362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:58.783645 23362 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:43999
I20260619 06:52:58.783711 23490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:43999 every 8 connection(s)
I20260619 06:52:58.784704 23362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:58.793390 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23362
I20260619 06:52:58.793515 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:58.793818 22980 external_mini_cluster.cc:1428] Setting key 909b58eeac3d6a06ba9f64244049bed3
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:52:58.802227 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.787156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:47645 (local address 127.22.113.62:41585)
0619 06:52:58.787369 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:58.787373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:58.788366 (+ 993us) server_negotiation.cc:407] Connection header received
0619 06:52:58.790039 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:58.790043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:58.790115 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:58.790239 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:58.791957 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.792600 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.794108 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.794374 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.794509 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.798126 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.798129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.798132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.798160 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.799811 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.800418 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.800428 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.800429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.800513 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.800912 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.800916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.800918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.801141 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.801264 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.801874 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.802030 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:52:58.803467 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.787568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41585 (local address 127.22.113.3:47645)
0619 06:52:58.788152 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:58.788208 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:58.789768 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:58.790463 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:58.790471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:58.791035 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:58.791769 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.791786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.792747 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.792760 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.793973 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:58.793985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.794563 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.794567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:58.794805 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.795673 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:58.795707 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:58.797888 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:58.799945 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.799952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.799972 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.800291 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.800621 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:58.800624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:58.800626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:58.800800 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:58.801259 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:58.801266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:58.801589 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 06:52:58.801819 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 06:52:58.802221 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":124,"threads_started":1}
I20260619 06:52:58.805073 23491 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41585
I20260619 06:52:58.805337 23491 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:58.805899 23491 heartbeater.cc:511] Master 127.22.113.62:41585 requested a full tablet report, sending...
I20260619 06:52:58.807143 23032 ts_manager.cc:194] Registered new tserver with Master: 48647e45b9d2432890dc30839fa517f0 (127.22.113.3:43999)
I20260619 06:52:58.807838 23032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:47645
I20260619 06:52:58.809602 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:52:58.839632 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.827966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.22.113.1:42385)
0619 06:52:58.828328 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:58.828339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:58.828465 (+ 126us) server_negotiation.cc:407] Connection header received
0619 06:52:58.828689 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:58.828696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:58.828939 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:58.829182 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:58.830178 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.831080 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.832171 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.832647 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.832793 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.835685 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.835696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.835714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.835743 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.837433 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.837953 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.837960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.837961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.838051 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.838366 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.838371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.838374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.838682 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.838939 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.839185 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.839397 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
W20260619 06:52:58.840891 23137 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60392
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:52:58.870302 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:58.860902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.22.113.1:42385)
0619 06:52:58.861113 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:58.861118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:58.861271 (+ 153us) server_negotiation.cc:407] Connection header received
0619 06:52:58.861473 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:58.861477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:58.861532 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:58.861629 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:58.862405 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.862966 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.863920 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:58.864166 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:58.864284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:58.867104 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:58.867108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:58.867110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:58.867135 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:58.868560 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.868999 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.869004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.869006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.869095 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:58.869378 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:58.869382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:58.869384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:58.869577 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:58.869725 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:58.869968 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:52:58.870114 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:52:58.871784 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23094
I20260619 06:52:58.877852 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23228
I20260619 06:52:58.884578 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23362
I20260619 06:52:58.890910 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23000
2026-06-19T06:52:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2360 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[23505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[23505](info): set up 2 sockets
Jun 19 06:52:58 dist-test-slave-g8zg krb5kdc[23505](info): commencing operation
krb5kdc: starting...
W20260619 06:52:59.938987 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:00.074106 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41133
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:41133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:00.203555 23518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:00.203879 23518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:00.203928 23518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:00.208115 23518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:00.208179 23518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:00.208200 23518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:00.208218 23518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:00.208284 23518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:00.214179 23518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:41133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:00.215543 23518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:00.216780 23518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:00.226903 23524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:00.227034 23518 server_base.cc:1034] running on GCE node
W20260619 06:53:00.226966 23523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:00.226981 23526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:00.227978 23518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:00.229353 23518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:00.230587 23518 hybrid_clock.cc:648] HybridClock initialized: now 1781851980230570 us; error 36 us; skew 500 ppm
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:00.233738 23518 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:00.234797 23518 webserver.cc:460] Webserver started at http://127.22.113.62:42089/ using document root <none> and password file <none>
I20260619 06:53:00.235509 23518 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:00.235608 23518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:00.235862 23518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:00.238157 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "2980d87573d949cabe9cd24002703d7b"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "58dd5bf470ee59e81865c667656ad91b"
server_key_iv: "c15575c24fed68b1ce40559fd5baac10"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.238765 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "2980d87573d949cabe9cd24002703d7b"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "58dd5bf470ee59e81865c667656ad91b"
server_key_iv: "c15575c24fed68b1ce40559fd5baac10"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.242874 23518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:00.245669 23533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.247187 23518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:00.247416 23518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "2980d87573d949cabe9cd24002703d7b"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "58dd5bf470ee59e81865c667656ad91b"
server_key_iv: "c15575c24fed68b1ce40559fd5baac10"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.247542 23518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:00.257956 23518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:00.260919 23518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:00.261116 23518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:00.268725 23518 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:41133
I20260619 06:53:00.268787 23585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:41133 every 8 connection(s)
I20260619 06:53:00.269714 23518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:00.272022 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23518
I20260619 06:53:00.272156 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:00.272468 22980 external_mini_cluster.cc:1428] Setting key 72f771de5ac473c2324fec4d4f40f331
I20260619 06:53:00.273561 23586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:00.280344 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b: Bootstrap starting.
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:00.283362 23586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:00.284308 23586 log.cc:824] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:00.286450 23586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b: No bootstrap required, opened a new log
I20260619 06:53:00.287688 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:00.274195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.22.113.62:41133)
0619 06:53:00.274736 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:00.274753 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:00.274789 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:53:00.275558 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:00.275576 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:00.275913 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:00.276256 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:00.277034 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.278034 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.278880 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.279443 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.279612 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:00.282731 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:00.282736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:00.282753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:00.282781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:00.284701 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.285147 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.285153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.285157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.285275 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.285552 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.285557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.285559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.285971 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:00.286163 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:00.286469 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:53:00.286792 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":191,"threads_started":1}
I20260619 06:53:00.290287 23586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } }
I20260619 06:53:00.290603 23586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:00.290661 23586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2980d87573d949cabe9cd24002703d7b, State: Initialized, Role: FOLLOWER
I20260619 06:53:00.291170 23586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } }
I20260619 06:53:00.291383 23586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:00.291471 23586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:00.291579 23586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:00.292814 23586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } }
I20260619 06:53:00.293210 23586 leader_election.cc:302] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2980d87573d949cabe9cd24002703d7b; no voters:
I20260619 06:53:00.293628 23586 leader_election.cc:288] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:00.293864 23591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:00.294184 23591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [term 1 LEADER]: Becoming Leader. State: Replica: 2980d87573d949cabe9cd24002703d7b, State: Running, Role: LEADER
I20260619 06:53:00.294613 23591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } }
I20260619 06:53:00.295173 23586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:00.296595 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2980d87573d949cabe9cd24002703d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } } }
I20260619 06:53:00.296756 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:00.296811 23593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2980d87573d949cabe9cd24002703d7b. Latest consensus state: current_term: 1 leader_uuid: "2980d87573d949cabe9cd24002703d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2980d87573d949cabe9cd24002703d7b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41133 } } }
I20260619 06:53:00.296934 23593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:00.298032 23600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:00.301852 23600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:00.308365 23600 catalog_manager.cc:1347] Generated new cluster ID: 31f95639260343dfa6e436e347eb4e7b
I20260619 06:53:00.308473 23600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:00.336494 23600 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:00.337551 23600 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:00.346725 23600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2980d87573d949cabe9cd24002703d7b: Generated new TSK 0
I20260619 06:53:00.347482 23600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:00.380491 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:00.504415 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:00.504668 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:00.504712 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:00.508739 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:00.508795 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:00.508891 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:00.514161 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:00.515178 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:00.516225 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:00.526553 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:00.526564 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:00.526566 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:00.526660 23612 server_base.cc:1034] running on GCE node
I20260619 06:53:00.527354 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:00.528014 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:00.529181 23612 hybrid_clock.cc:648] HybridClock initialized: now 1781851980529159 us; error 38 us; skew 500 ppm
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:00.532284 23612 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:00.533380 23612 webserver.cc:460] Webserver started at http://127.22.113.1:36711/ using document root <none> and password file <none>
I20260619 06:53:00.534016 23612 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:00.534081 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:00.534284 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:00.536695 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "25948d3d781042a190d715c2a2810429"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "b97865c4a3d0db5a66aeafd30436eb68"
server_key_iv: "6c816838fbd8ed6cf6b2fa49533d74c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.537350 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "25948d3d781042a190d715c2a2810429"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "b97865c4a3d0db5a66aeafd30436eb68"
server_key_iv: "6c816838fbd8ed6cf6b2fa49533d74c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.541777 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:00.544661 23627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.546283 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:00.546435 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "25948d3d781042a190d715c2a2810429"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "b97865c4a3d0db5a66aeafd30436eb68"
server_key_iv: "6c816838fbd8ed6cf6b2fa49533d74c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.546569 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:00.578364 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:00.582123 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:00.582353 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:00.583069 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:00.584088 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:00.584164 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.584235 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:00.584280 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.595359 23612 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:42537
I20260619 06:53:00.595428 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:42537 every 8 connection(s)
I20260619 06:53:00.596644 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:00.598388 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23612
I20260619 06:53:00.598531 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:00.598850 22980 external_mini_cluster.cc:1428] Setting key 93524fee89faf1704c8485f92e1cc142
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:00.612339 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:00.598865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:33279 (local address 127.22.113.62:41133)
0619 06:53:00.599056 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:00.599060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:00.600385 (+ 1325us) server_negotiation.cc:407] Connection header received
0619 06:53:00.601817 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:00.601820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:00.601877 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:00.601954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:00.603197 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.603810 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.604981 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.605223 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.605334 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:00.608417 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:00.608421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:00.608423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:00.608445 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:00.610067 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.610665 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.610670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.610671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.610751 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.611106 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.611111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.611113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.611355 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:00.611480 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:00.612027 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 06:53:00.612157 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:53:00.613157 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:00.599476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41133 (local address 127.22.113.1:33279)
0619 06:53:00.600148 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:00.600211 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:00.601574 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:00.602135 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:00.602141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:00.602546 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:00.603040 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:00.603053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.603956 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.603960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:00.604853 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:00.604862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.605380 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.605384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:00.605548 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:00.606341 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:00.606362 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:00.608200 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:00.610216 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:00.610222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:00.610238 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:00.610556 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:00.610849 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:00.610852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:00.610854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:00.611008 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:00.611478 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:00.611485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:00.611732 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 06:53:00.611974 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 06:53:00.612294 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":160,"threads_started":1}
I20260619 06:53:00.614877 23741 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41133
I20260619 06:53:00.615167 23741 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:00.615703 23741 heartbeater.cc:511] Master 127.22.113.62:41133 requested a full tablet report, sending...
I20260619 06:53:00.617738 23550 ts_manager.cc:194] Registered new tserver with Master: 25948d3d781042a190d715c2a2810429 (127.22.113.1:42537)
I20260619 06:53:00.619728 23550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:33279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:00.631317 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:00.754160 23746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:00.754459 23746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:00.754506 23746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:00.758760 23746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:00.758824 23746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:00.758888 23746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:00.764571 23746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:00.765796 23746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:00.766939 23746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:00.778024 23751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:00.778143 23746 server_base.cc:1034] running on GCE node
W20260619 06:53:00.778024 23752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:00.778450 23754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:00.778898 23746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:00.779717 23746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:00.780910 23746 hybrid_clock.cc:648] HybridClock initialized: now 1781851980780893 us; error 35 us; skew 500 ppm
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:00.784017 23746 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:00.785008 23746 webserver.cc:460] Webserver started at http://127.22.113.2:38193/ using document root <none> and password file <none>
I20260619 06:53:00.785632 23746 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:00.785701 23746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:00.785885 23746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:00.788269 23746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "2f81a92836044b0eaf3b08b8fa935d13"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "0b49976fe15f6c5b9d401c81542beafa"
server_key_iv: "3a6e74ffa8addd13cc6b49cc2884d9dd"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.788831 23746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "2f81a92836044b0eaf3b08b8fa935d13"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "0b49976fe15f6c5b9d401c81542beafa"
server_key_iv: "3a6e74ffa8addd13cc6b49cc2884d9dd"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.792948 23746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:00.796075 23761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.797814 23746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:00.797999 23746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "2f81a92836044b0eaf3b08b8fa935d13"
format_stamp: "Formatted at 2026-06-19 06:53:00 on dist-test-slave-g8zg"
server_key: "0b49976fe15f6c5b9d401c81542beafa"
server_key_iv: "3a6e74ffa8addd13cc6b49cc2884d9dd"
server_key_version: "encryptionkey@0"
I20260619 06:53:00.798132 23746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:00.821902 23746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:00.825670 23746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:00.825894 23746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:00.826618 23746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:00.827591 23746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:00.827669 23746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.827765 23746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:00.827816 23746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:00.838778 23746 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:39895
I20260619 06:53:00.838809 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:39895 every 8 connection(s)
I20260619 06:53:00.839870 23746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:00.848874 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23746
I20260619 06:53:00.849030 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:00.849331 22980 external_mini_cluster.cc:1428] Setting key 2163bd45cb754671b76a36ab7e01c0d0
Jun 19 06:53:00 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:00.856022 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:00.842001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:47281 (local address 127.22.113.62:41133)
0619 06:53:00.842176 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:00.842181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:00.843155 (+ 974us) server_negotiation.cc:407] Connection header received
0619 06:53:00.844660 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:00.844663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:00.844723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:00.844828 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:00.846108 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.846771 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.848025 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.848287 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.848415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:00.851703 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:00.851709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:00.851715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:00.851748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:00.853429 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.854093 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.854096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.854097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.854173 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:00.854614 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:00.854621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:00.854628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:00.854867 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:00.854987 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:00.855682 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 06:53:00.855829 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:53:00.856988 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:00.842427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41133 (local address 127.22.113.2:47281)
0619 06:53:00.842968 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:00.843021 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:00.844415 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:00.845039 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:00.845047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:00.845438 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:00.845964 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:00.845976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.846902 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.846911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:00.847895 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:00.847905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:00.848472 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:00.848476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:00.848674 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:00.849476 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:00.849505 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:00.851517 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:00.853564 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:00.853572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:00.853591 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:00.853974 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:00.854266 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:00.854271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:00.854274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:00.854469 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:00.854992 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:00.855002 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:00.855408 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 06:53:00.855613 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:53:00.855992 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260619 06:53:00.858336 23875 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41133
I20260619 06:53:00.858621 23875 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:00.859210 23875 heartbeater.cc:511] Master 127.22.113.62:41133 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:00.860486 23550 ts_manager.cc:194] Registered new tserver with Master: 2f81a92836044b0eaf3b08b8fa935d13 (127.22.113.2:39895)
I20260619 06:53:00.861157 23550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:47281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:00.879107 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:00.998347 23880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:00.998652 23880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:00.998700 23880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:01.002938 23880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:01.003016 23880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:01.003080 23880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:01.008513 23880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41133
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:01.009591 23880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:01.010607 23880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:01.021150 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:01.021144 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:01.021169 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:01.021585 23880 server_base.cc:1034] running on GCE node
I20260619 06:53:01.022141 23880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:01.022958 23880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:01.024155 23880 hybrid_clock.cc:648] HybridClock initialized: now 1781851981024119 us; error 52 us; skew 500 ppm
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:01.027493 23880 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:01.028518 23880 webserver.cc:460] Webserver started at http://127.22.113.3:41935/ using document root <none> and password file <none>
I20260619 06:53:01.029191 23880 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:01.029284 23880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:01.029551 23880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:01.032127 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "e7288835d953489f8546992f32e90cb3"
format_stamp: "Formatted at 2026-06-19 06:53:01 on dist-test-slave-g8zg"
server_key: "7e087571d36c1ac4e4b73d8e693cbd74"
server_key_iv: "4dbe158714e3c79519beff0bb78ce7b9"
server_key_version: "encryptionkey@0"
I20260619 06:53:01.032759 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "e7288835d953489f8546992f32e90cb3"
format_stamp: "Formatted at 2026-06-19 06:53:01 on dist-test-slave-g8zg"
server_key: "7e087571d36c1ac4e4b73d8e693cbd74"
server_key_iv: "4dbe158714e3c79519beff0bb78ce7b9"
server_key_version: "encryptionkey@0"
I20260619 06:53:01.037207 23880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:01.040099 23895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:01.041549 23880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:01.041702 23880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "e7288835d953489f8546992f32e90cb3"
format_stamp: "Formatted at 2026-06-19 06:53:01 on dist-test-slave-g8zg"
server_key: "7e087571d36c1ac4e4b73d8e693cbd74"
server_key_iv: "4dbe158714e3c79519beff0bb78ce7b9"
server_key_version: "encryptionkey@0"
I20260619 06:53:01.041832 23880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:01.057122 23880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:01.060757 23880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:01.060972 23880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:01.061659 23880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:01.062588 23880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:01.062668 23880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:01.062757 23880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:01.062799 23880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:01.073469 23880 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:33919
I20260619 06:53:01.073560 24008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:33919 every 8 connection(s)
I20260619 06:53:01.074548 23880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:01.076741 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 23880
I20260619 06:53:01.076879 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:01.077198 22980 external_mini_cluster.cc:1428] Setting key 54225f5bf94630eece9d17a44316975e
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:01.088805 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.076714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:58807 (local address 127.22.113.62:41133)
0619 06:53:01.076912 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.076917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.077894 (+ 977us) server_negotiation.cc:407] Connection header received
0619 06:53:01.079045 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.079049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.079119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.079224 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:01.080419 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.080952 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.082097 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.082313 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.082438 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.085226 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:01.085229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:01.085231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:01.085255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:01.086630 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.087177 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.087179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.087181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.087325 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.087690 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.087693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.087694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.087896 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:01.087992 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.088552 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.088663 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:53:01.090001 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.077112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41133 (local address 127.22.113.3:58807)
0619 06:53:01.077674 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.077744 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.078783 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.079421 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.079428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.079798 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:01.080274 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.080286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.081078 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.081084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.081960 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.081970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.082470 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.082475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.082649 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.083397 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:01.083417 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:01.085017 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:01.086774 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:01.086779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:01.086793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:01.087058 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:01.087434 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:01.087436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:01.087437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:01.087577 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:01.088000 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:01.088005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:01.088297 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.088475 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.088785 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":182,"threads_started":1}
I20260619 06:53:01.091645 24009 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41133
I20260619 06:53:01.091955 24009 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:01.092516 24009 heartbeater.cc:511] Master 127.22.113.62:41133 requested a full tablet report, sending...
I20260619 06:53:01.093693 23550 ts_manager.cc:194] Registered new tserver with Master: e7288835d953489f8546992f32e90cb3 (127.22.113.3:33919)
I20260619 06:53:01.094231 23550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:58807
I20260619 06:53:01.103832 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:53:01.114903 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.107414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.22.113.62:41133)
0619 06:53:01.107557 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.107561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.107813 (+ 252us) server_negotiation.cc:407] Connection header received
0619 06:53:01.107972 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.107975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.108023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.108110 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:01.108885 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.109414 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.110269 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.110491 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.110571 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.112067 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:01.112070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:01.112072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:01.112093 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:01.113390 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.113802 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.113804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.113805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.113876 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.114144 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.114146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.114147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.114372 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:01.114511 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.114691 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.114783 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:53:01.120218 23550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:01.123370 23550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:01.139688 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.134088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.22.113.3:33919)
0619 06:53:01.134462 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.134475 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.134674 (+ 199us) server_negotiation.cc:407] Connection header received
0619 06:53:01.134781 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.134787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.135008 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.135167 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:01.135690 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.136742 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.138079 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.138633 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.138758 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.138905 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.139397 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.139504 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260619 06:53:01.140280 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.133196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:39895 (local address 127.0.0.1:42550)
0619 06:53:01.134507 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.134522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.134632 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.135313 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.135316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.135333 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:01.135528 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.135536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.136997 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.137000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.138607 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.138613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.139725 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.139728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.139811 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.139835 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.139910 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.140058 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"mutex_wait_us":984,"spinlock_wait_cycles":9216,"thread_start_us":144,"threads_started":1}
I20260619 06:53:01.140569 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.134039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.22.113.1:42537)
0619 06:53:01.134696 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.134703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.134731 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:53:01.134805 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.134809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.134974 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.135166 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:01.135518 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.136827 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.138747 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.139661 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.139777 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.139933 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.140264 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.140403 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":116,"threads_started":1}
I20260619 06:53:01.140777 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.133386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.22.113.2:39895)
0619 06:53:01.134352 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.134369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.134716 (+ 347us) server_negotiation.cc:407] Connection header received
0619 06:53:01.134793 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.134797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.135008 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.135175 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:01.135724 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.136836 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.138737 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.139403 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.139506 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.139667 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.140506 (+ 839us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.140611 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":211,"threads_started":1}
I20260619 06:53:01.140336 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.133577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33919 (local address 127.0.0.1:32922)
0619 06:53:01.134479 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.134496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.134633 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.135367 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.135370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.135393 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:01.135566 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.135571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.136874 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.136880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.137936 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.137946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.138980 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.138983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.139106 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.139142 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.140066 (+ 924us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.140158 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":125,"spinlock_wait_cycles":11520,"thread_start_us":117,"threads_started":1}
I20260619 06:53:01.140497 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.133196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:42537 (local address 127.0.0.1:40882)
0619 06:53:01.134255 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.134310 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.134464 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.135171 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.135177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.135200 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:01.135375 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.135382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.137001 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.137004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.138593 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.138606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.140001 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.140004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.140107 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.140129 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.140211 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.140284 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":156,"thread_start_us":89,"threads_started":1}
I20260619 06:53:01.144296 23809 tablet_service.cc:1467] Processing CreateTablet for tablet c550aff9514c41e780f909070a0ce567 (DEFAULT_TABLE table=test-table [id=e663d8300f724afab4c4ed27aa049194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:01.144289 23675 tablet_service.cc:1467] Processing CreateTablet for tablet c550aff9514c41e780f909070a0ce567 (DEFAULT_TABLE table=test-table [id=e663d8300f724afab4c4ed27aa049194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:01.144289 23943 tablet_service.cc:1467] Processing CreateTablet for tablet c550aff9514c41e780f909070a0ce567 (DEFAULT_TABLE table=test-table [id=e663d8300f724afab4c4ed27aa049194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:01.145424 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c550aff9514c41e780f909070a0ce567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:01.145459 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c550aff9514c41e780f909070a0ce567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:01.145459 23943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c550aff9514c41e780f909070a0ce567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:01.154340 24027 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Bootstrap starting.
I20260619 06:53:01.154340 24026 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Bootstrap starting.
I20260619 06:53:01.156858 24027 tablet_bootstrap.cc:654] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:01.157306 24026 tablet_bootstrap.cc:654] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:01.157840 24027 log.cc:824] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:01.158041 24026 log.cc:824] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:01.158636 24028 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Bootstrap starting.
I20260619 06:53:01.160251 24027 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: No bootstrap required, opened a new log
I20260619 06:53:01.160410 24026 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: No bootstrap required, opened a new log
I20260619 06:53:01.160524 24027 ts_tablet_manager.cc:1397] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 06:53:01.160749 24026 ts_tablet_manager.cc:1397] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:53:01.160943 24028 tablet_bootstrap.cc:654] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:01.162155 24028 log.cc:824] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:01.164517 24028 tablet_bootstrap.cc:492] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: No bootstrap required, opened a new log
I20260619 06:53:01.164743 24028 ts_tablet_manager.cc:1397] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:53:01.164773 24027 raft_consensus.cc:348] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.165055 24027 raft_consensus.cc:374] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:01.164860 24026 raft_consensus.cc:348] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.165129 24027 raft_consensus.cc:729] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25948d3d781042a190d715c2a2810429, State: Initialized, Role: FOLLOWER
I20260619 06:53:01.165191 24026 raft_consensus.cc:374] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:01.165298 24026 raft_consensus.cc:729] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f81a92836044b0eaf3b08b8fa935d13, State: Initialized, Role: FOLLOWER
I20260619 06:53:01.165736 24027 consensus_queue.cc:260] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.165776 24026 consensus_queue.cc:260] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.166555 23741 heartbeater.cc:503] Master 127.22.113.62:41133 was elected leader, sending a full tablet report...
I20260619 06:53:01.166828 24026 ts_tablet_manager.cc:1428] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:53:01.166864 24027 ts_tablet_manager.cc:1428] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:53:01.167308 23875 heartbeater.cc:503] Master 127.22.113.62:41133 was elected leader, sending a full tablet report...
I20260619 06:53:01.169785 24028 raft_consensus.cc:348] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.170112 24028 raft_consensus.cc:374] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:01.170353 24028 raft_consensus.cc:729] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7288835d953489f8546992f32e90cb3, State: Initialized, Role: FOLLOWER
I20260619 06:53:01.170899 24028 consensus_queue.cc:260] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.171661 24009 heartbeater.cc:503] Master 127.22.113.62:41133 was elected leader, sending a full tablet report...
I20260619 06:53:01.172081 24028 ts_tablet_manager.cc:1428] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:53:01.321609 24034 raft_consensus.cc:482] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:01.321861 24034 raft_consensus.cc:504] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.323318 24034 leader_election.cc:288] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f81a92836044b0eaf3b08b8fa935d13 (127.22.113.2:39895), 25948d3d781042a190d715c2a2810429 (127.22.113.1:42537)
W20260619 06:53:01.325773 24010 tablet.cc:2369] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:01.327925 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.323584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:39895 (local address 127.22.113.3:34037)
0619 06:53:01.323812 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.323830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.323941 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.324342 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.324345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.324363 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:01.324535 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.324542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.325475 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.325479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.326576 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.326585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.327412 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.327416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.327537 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.327555 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.327659 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.327752 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1}
I20260619 06:53:01.328008 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.323758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:34037 (local address 127.22.113.2:39895)
0619 06:53:01.323896 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.323901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.323918 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:01.324121 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.324127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.324205 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.324296 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:01.324661 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.325257 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.326706 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.327217 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.327398 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.327456 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.327717 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.327851 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:53:01.328087 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.323891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:47921 (local address 127.22.113.1:42537)
0619 06:53:01.324086 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.324089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.324210 (+ 121us) server_negotiation.cc:407] Connection header received
0619 06:53:01.324346 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.324349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.324407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.324502 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:01.324835 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.325510 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.327037 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.327539 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.327686 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.327744 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.327887 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.327958 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:53:01.328094 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.323667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:42537 (local address 127.22.113.3:47921)
0619 06:53:01.324093 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:01.324108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:01.324224 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:01.324521 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:01.324524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:01.324538 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:01.324705 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.324712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.325650 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.325653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.326906 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:01.326917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.327664 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.327668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:01.327754 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.327773 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:53:01.327862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:53:01.327918 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":43,"thread_start_us":157,"threads_started":1}
I20260619 06:53:01.328845 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c550aff9514c41e780f909070a0ce567" candidate_uuid: "e7288835d953489f8546992f32e90cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f81a92836044b0eaf3b08b8fa935d13" is_pre_election: true
I20260619 06:53:01.328866 23695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c550aff9514c41e780f909070a0ce567" candidate_uuid: "e7288835d953489f8546992f32e90cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25948d3d781042a190d715c2a2810429" is_pre_election: true
I20260619 06:53:01.329200 23829 raft_consensus.cc:2454] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7288835d953489f8546992f32e90cb3 in term 0.
I20260619 06:53:01.329200 23695 raft_consensus.cc:2454] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7288835d953489f8546992f32e90cb3 in term 0.
I20260619 06:53:01.329756 23897 leader_election.cc:302] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25948d3d781042a190d715c2a2810429, 2f81a92836044b0eaf3b08b8fa935d13, e7288835d953489f8546992f32e90cb3; no voters:
I20260619 06:53:01.330041 24034 raft_consensus.cc:2781] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:01.330148 24034 raft_consensus.cc:482] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:01.330199 24034 raft_consensus.cc:3037] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:01.331588 24034 raft_consensus.cc:504] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.332015 24034 leader_election.cc:288] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 2f81a92836044b0eaf3b08b8fa935d13 (127.22.113.2:39895), 25948d3d781042a190d715c2a2810429 (127.22.113.1:42537)
I20260619 06:53:01.332372 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c550aff9514c41e780f909070a0ce567" candidate_uuid: "e7288835d953489f8546992f32e90cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f81a92836044b0eaf3b08b8fa935d13"
I20260619 06:53:01.332472 23695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c550aff9514c41e780f909070a0ce567" candidate_uuid: "e7288835d953489f8546992f32e90cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25948d3d781042a190d715c2a2810429"
I20260619 06:53:01.332507 23829 raft_consensus.cc:3037] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:01.332600 23695 raft_consensus.cc:3037] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:01.333977 23695 raft_consensus.cc:2454] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7288835d953489f8546992f32e90cb3 in term 1.
I20260619 06:53:01.333977 23829 raft_consensus.cc:2454] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7288835d953489f8546992f32e90cb3 in term 1.
I20260619 06:53:01.334331 23897 leader_election.cc:302] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25948d3d781042a190d715c2a2810429, e7288835d953489f8546992f32e90cb3; no voters:
I20260619 06:53:01.334580 24034 raft_consensus.cc:2781] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:01.334820 24034 raft_consensus.cc:686] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [term 1 LEADER]: Becoming Leader. State: Replica: e7288835d953489f8546992f32e90cb3, State: Running, Role: LEADER
I20260619 06:53:01.335294 24034 consensus_queue.cc:237] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } }
I20260619 06:53:01.338925 23549 catalog_manager.cc:5495] T c550aff9514c41e780f909070a0ce567 P e7288835d953489f8546992f32e90cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7288835d953489f8546992f32e90cb3 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "e7288835d953489f8546992f32e90cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f81a92836044b0eaf3b08b8fa935d13" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25948d3d781042a190d715c2a2810429" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7288835d953489f8546992f32e90cb3" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33919 } health_report { overall_health: HEALTHY } } }
W20260619 06:53:01.341104 23876 tablet.cc:2369] T c550aff9514c41e780f909070a0ce567 P 2f81a92836044b0eaf3b08b8fa935d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:01.348099 23742 tablet.cc:2369] T c550aff9514c41e780f909070a0ce567 P 25948d3d781042a190d715c2a2810429: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:01.376669 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.367498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.22.113.1:42537)
0619 06:53:01.367690 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.367697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.367866 (+ 169us) server_negotiation.cc:407] Connection header received
0619 06:53:01.367988 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.367991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.368046 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.368125 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:01.368799 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.369353 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.370225 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.370438 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.370515 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.373022 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:01.373033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:01.373052 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:01.373077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:01.374993 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.375445 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.375451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.375452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.375532 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.375751 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.375753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.375754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.376005 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:01.376194 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.376315 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.376453 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:01.405624 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.396453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.22.113.1:42537)
0619 06:53:01.396630 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.396634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.396852 (+ 218us) server_negotiation.cc:407] Connection header received
0619 06:53:01.397034 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.397041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.397086 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.397176 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:01.397929 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.398468 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.399340 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.399573 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.399685 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.402093 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:01.402097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:01.402099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:01.402129 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:01.403707 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.404257 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.404261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.404262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.404339 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.404638 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.404643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.404645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.404825 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:01.405000 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.405301 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.405434 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[23505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:01.435437 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:01.425977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.22.113.1:42537)
0619 06:53:01.426196 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:01.426201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:01.426375 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:53:01.426571 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:01.426577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:01.426632 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:01.426728 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:01.427505 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.428041 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.428908 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:01.429150 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:01.429264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:01.432188 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:01.432192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:01.432194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:01.432215 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:01.433697 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.434122 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.434127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.434129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.434216 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:01.434485 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:01.434489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:01.434491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:01.434682 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:01.434821 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:01.435064 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:53:01.435204 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:53:01.437039 23655 tablet_service.cc:2908] Rejecting scan request for tablet c550aff9514c41e780f909070a0ce567: Uninitialized: clean time has not yet been initialized
I20260619 06:53:01.439541 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23612
I20260619 06:53:01.446310 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23746
I20260619 06:53:01.453001 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23880
I20260619 06:53:01.459558 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 23518
2026-06-19T06:53:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2569 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[24050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[24050](info): set up 2 sockets
Jun 19 06:53:01 dist-test-slave-g8zg krb5kdc[24050](info): commencing operation
krb5kdc: starting...
W20260619 06:53:02.535175 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:02 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:02.668941 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42903
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:02.792609 24063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:02.793030 24063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:02.793123 24063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:02.797606 24063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:02.797720 24063 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:53:02.797753 24063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:02.797798 24063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:02.797838 24063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:02.797894 24063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:02.804051 24063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:42903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:02.805459 24063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:02.806725 24063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:02.816485 24068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:02.816555 24071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:02.816660 24063 server_base.cc:1034] running on GCE node
W20260619 06:53:02.816727 24069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:02.817448 24063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:02.818749 24063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:02.819959 24063 hybrid_clock.cc:648] HybridClock initialized: now 1781851982819915 us; error 59 us; skew 500 ppm
Jun 19 06:53:02 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:02.826483 24063 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:02.827718 24063 webserver.cc:460] Webserver started at http://127.22.113.62:36885/ using document root <none> and password file <none>
I20260619 06:53:02.828413 24063 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:02.828510 24063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:02.828784 24063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:02.831167 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "4887613610414116abff24f95b6a0c25"
format_stamp: "Formatted at 2026-06-19 06:53:02 on dist-test-slave-g8zg"
server_key: "756e7c9f05838f33b14d1a487c346718"
server_key_iv: "2f76177c361bc79b677032d48f744abc"
server_key_version: "encryptionkey@0"
I20260619 06:53:02.831892 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "4887613610414116abff24f95b6a0c25"
format_stamp: "Formatted at 2026-06-19 06:53:02 on dist-test-slave-g8zg"
server_key: "756e7c9f05838f33b14d1a487c346718"
server_key_iv: "2f76177c361bc79b677032d48f744abc"
server_key_version: "encryptionkey@0"
I20260619 06:53:02.836679 24063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:02.839918 24078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:02.841609 24063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:02.841804 24063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "4887613610414116abff24f95b6a0c25"
format_stamp: "Formatted at 2026-06-19 06:53:02 on dist-test-slave-g8zg"
server_key: "756e7c9f05838f33b14d1a487c346718"
server_key_iv: "2f76177c361bc79b677032d48f744abc"
server_key_version: "encryptionkey@0"
I20260619 06:53:02.841949 24063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:02.857405 24063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:02.860738 24063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:02.860983 24063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:02.869760 24063 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:42903
I20260619 06:53:02.869906 24140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:42903 every 8 connection(s)
I20260619 06:53:02.870853 24063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:02.874342 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:02.876802 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24063
I20260619 06:53:02.876935 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:02.877241 22980 external_mini_cluster.cc:1428] Setting key 5f4456b52fa9a5199b673062561e4d32
I20260619 06:53:02.882474 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Bootstrap starting.
I20260619 06:53:02.885377 24141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:02.886260 24141 log.cc:824] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:53:02 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:02.889056 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: No bootstrap required, opened a new log
I20260619 06:53:02.893752 24141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } }
I20260619 06:53:02.894058 24141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:02.894116 24141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4887613610414116abff24f95b6a0c25, State: Initialized, Role: FOLLOWER
I20260619 06:53:02.894456 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:02.879020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.22.113.62:42903)
0619 06:53:02.879647 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:02.879667 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:02.879712 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:53:02.880557 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:02.880577 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:02.880964 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:02.881324 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:02.882166 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:02.883299 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:02.884173 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:02.884666 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:02.884851 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:02.887830 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:02.887837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:02.887856 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:02.887891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:02.890715 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:02.891271 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:02.891278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:02.891285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:02.891433 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:02.891738 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:02.891741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:02.891744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:02.892333 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:02.892546 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:02.892864 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:53:02.893252 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":190,"threads_started":1}
I20260619 06:53:02.894696 24141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } }
I20260619 06:53:02.894860 24141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:02.894929 24141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:02.895031 24141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:02.896255 24141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } }
I20260619 06:53:02.896658 24141 leader_election.cc:302] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4887613610414116abff24f95b6a0c25; no voters:
I20260619 06:53:02.897042 24141 leader_election.cc:288] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:02.897212 24146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:02.897552 24146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [term 1 LEADER]: Becoming Leader. State: Replica: 4887613610414116abff24f95b6a0c25, State: Running, Role: LEADER
I20260619 06:53:02.897927 24146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } }
I20260619 06:53:02.898932 24141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:02.899820 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4887613610414116abff24f95b6a0c25. Latest consensus state: current_term: 1 leader_uuid: "4887613610414116abff24f95b6a0c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } } }
I20260619 06:53:02.899979 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:02.900434 24152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:02.901302 24147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4887613610414116abff24f95b6a0c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4887613610414116abff24f95b6a0c25" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42903 } } }
I20260619 06:53:02.901415 24147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:02.904147 24152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:02.911141 24152 catalog_manager.cc:1347] Generated new cluster ID: a809fbd9916342cb8d673cc43223fc2c
I20260619 06:53:02.911239 24152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:02.940394 24152 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:02.941473 24152 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:02.945498 24152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Generated new TSK 0
I20260619 06:53:02.946620 24152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:02.990705 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42903
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:03.113461 24167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:03.113801 24167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:03.113891 24167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:03.118093 24167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:03.118202 24167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:03.118286 24167 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:03.118376 24167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:03.123975 24167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:03.125145 24167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:03.126247 24167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:03.137102 24172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.137123 24173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.137102 24175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:03.137467 24167 server_base.cc:1034] running on GCE node
I20260619 06:53:03.138028 24167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:03.138736 24167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:03.139918 24167 hybrid_clock.cc:648] HybridClock initialized: now 1781851983139894 us; error 42 us; skew 500 ppm
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:03.143018 24167 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:03.144145 24167 webserver.cc:460] Webserver started at http://127.22.113.1:40247/ using document root <none> and password file <none>
I20260619 06:53:03.144821 24167 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:03.144922 24167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:03.145188 24167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:03.147616 24167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "9bb8d9cf4b0f4977a33eb97497b683fc"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "f2e9d16d025e7e5341c707399d988d5d"
server_key_iv: "d8a5dc292933da374a43b1075e28dc17"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.148164 24167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "9bb8d9cf4b0f4977a33eb97497b683fc"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "f2e9d16d025e7e5341c707399d988d5d"
server_key_iv: "d8a5dc292933da374a43b1075e28dc17"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.152333 24167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:03.155027 24182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.156764 24167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:03.156895 24167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "9bb8d9cf4b0f4977a33eb97497b683fc"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "f2e9d16d025e7e5341c707399d988d5d"
server_key_iv: "d8a5dc292933da374a43b1075e28dc17"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.156998 24167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:03.167891 24167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:03.171162 24167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:03.171424 24167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:03.173022 24167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:03.173075 24167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.173116 24167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:03.173180 24167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:03.184933 24167 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:44765
I20260619 06:53:03.185433 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:44765 every 8 connection(s)
I20260619 06:53:03.186050 24167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:03.188925 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24167
I20260619 06:53:03.189049 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:03.189179 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.174262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:58415 (local address 127.22.113.62:42903)
0619 06:53:03.174461 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.174467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.175458 (+ 991us) server_negotiation.cc:407] Connection header received
0619 06:53:03.176814 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.176818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.176886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.177012 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.178847 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.179844 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.181259 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.181520 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.181633 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.185097 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.185100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.185103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.185130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.186808 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.187438 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.187442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.187444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.187569 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.187905 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.187908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.187909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.188111 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.188197 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.188753 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.188992 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:53:03.189374 22980 external_mini_cluster.cc:1428] Setting key d8c3fb47287454796bed2d13b7b2a777
I20260619 06:53:03.190253 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.174706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.1:58415)
0619 06:53:03.175239 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.175317 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.176538 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.177263 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.177274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.177805 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.178622 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.178647 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.179982 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.179987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.181132 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.181143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.181694 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.181698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.181942 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.182710 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.182747 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.184907 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.186942 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.186947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.186963 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.187301 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.187675 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.187678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.187680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.187810 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.188224 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.188229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.188487 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.188687 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.189066 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":166,"threads_started":1}
I20260619 06:53:03.191915 24298 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42903
I20260619 06:53:03.192094 24298 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:03.192752 24298 heartbeater.cc:511] Master 127.22.113.62:42903 requested a full tablet report, sending...
I20260619 06:53:03.194661 24093 ts_manager.cc:194] Registered new tserver with Master: 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765)
I20260619 06:53:03.196506 24093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:58415
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:03.203195 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.195429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44643 (local address 127.22.113.62:42903)
0619 06:53:03.195562 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.195566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.195577 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:53:03.195613 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.195615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.195651 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.195756 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.196243 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.196996 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.197840 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.198136 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.198247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.199549 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.199553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.199556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.199580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.201445 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.201947 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.201952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.201954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.202039 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.202325 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.202329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.202331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.202521 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.202626 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.202896 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.203004 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260619 06:53:03.203207 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.194343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.1:44643)
0619 06:53:03.194670 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.194685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.194802 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.195778 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.195782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.196021 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.196131 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.196137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.197105 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.197109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.197713 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.197721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.198256 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.198260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.198371 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.198947 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.198959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.199417 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.201557 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.201561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.201563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.201849 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.202140 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.202143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.202145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.202230 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.202626 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.202630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.202734 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.202842 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.203004 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:03.222807 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42903
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:03.346001 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:03.346352 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:03.346405 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:03.350517 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:03.350579 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:03.350617 24307 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:03.350652 24307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:03.356055 24307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:03.357256 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:03.358395 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:03.369755 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.369782 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.369782 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:03.370169 24307 server_base.cc:1034] running on GCE node
I20260619 06:53:03.370633 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:03.371464 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:03.372641 24307 hybrid_clock.cc:648] HybridClock initialized: now 1781851983372596 us; error 65 us; skew 500 ppm
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:03.375880 24307 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:03.376910 24307 webserver.cc:460] Webserver started at http://127.22.113.2:40593/ using document root <none> and password file <none>
I20260619 06:53:03.377627 24307 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:03.377719 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:03.377977 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:03.380448 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "6d254570b7c043efa2869969b6bc4e77"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "246a60425ee8dd01bfdc9ed14f3a9a7b"
server_key_iv: "ead4fd69403e6392dc5fdb516e24a2a6"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.381045 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "6d254570b7c043efa2869969b6bc4e77"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "246a60425ee8dd01bfdc9ed14f3a9a7b"
server_key_iv: "ead4fd69403e6392dc5fdb516e24a2a6"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.385283 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:53:03.388139 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.389621 24307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:53:03.389756 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "6d254570b7c043efa2869969b6bc4e77"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "246a60425ee8dd01bfdc9ed14f3a9a7b"
server_key_iv: "ead4fd69403e6392dc5fdb516e24a2a6"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.389864 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:03.407651 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:03.411073 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:03.411332 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:03.413076 24307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:03.413137 24307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.413216 24307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:03.413251 24307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:03.426222 24307 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:46183
I20260619 06:53:03.426363 24437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:46183 every 8 connection(s)
I20260619 06:53:03.427573 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:03.429518 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.414416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:58281 (local address 127.22.113.62:42903)
0619 06:53:03.414589 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.414594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.415722 (+ 1128us) server_negotiation.cc:407] Connection header received
0619 06:53:03.417470 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.417475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.417555 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.417672 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.419616 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.420258 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.421794 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.422042 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.422157 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.425128 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.425133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.425136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.425175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.427163 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.427812 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.427815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.427817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.427912 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.428336 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.428340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.428342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.428625 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.428734 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.429272 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.429394 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:53:03.430693 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.414865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.2:58281)
0619 06:53:03.415512 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.415576 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.417116 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.417912 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.417922 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.418512 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.419418 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.419441 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.420432 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.420437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.421658 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.421671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.422218 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.422223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.422425 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.423207 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.423225 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.424907 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.427335 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.427347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.427360 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.427665 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.428010 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.428014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.428017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.428220 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.428748 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.428754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.429025 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.429199 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.429501 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":142,"threads_started":1}
I20260619 06:53:03.430887 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24307
I20260619 06:53:03.430994 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:03.431427 22980 external_mini_cluster.cc:1428] Setting key 0e404a6874c2f72b95f6b4fb6510b051
I20260619 06:53:03.432710 24438 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42903
I20260619 06:53:03.432963 24438 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:03.433648 24438 heartbeater.cc:511] Master 127.22.113.62:42903 requested a full tablet report, sending...
I20260619 06:53:03.435894 24095 ts_manager.cc:194] Registered new tserver with Master: 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183)
I20260619 06:53:03.436699 24095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:58281
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:03.444185 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.435875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:55583 (local address 127.22.113.62:42903)
0619 06:53:03.436024 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.436032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.436203 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:53:03.436333 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.436337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.436396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.436485 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.437040 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.437743 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.438800 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.439105 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.439187 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.440696 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.440703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.440706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.440733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.442432 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.442893 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.442899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.442902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.442985 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.443328 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.443333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.443335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.443641 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.443737 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.443972 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.444066 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:53:03.444319 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.435758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.2:55583)
0619 06:53:03.436103 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.436118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.436225 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.436529 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.436532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.436764 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.436892 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.436901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.437898 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.437902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.438670 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.438678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.439397 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.439400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.439497 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.440140 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.440152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.440538 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.442565 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.442567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.442569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.442777 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.443083 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.443085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.443086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.443158 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.443744 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.443746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.443836 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.443922 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.444047 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:03.462736 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42903
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:03.583678 24447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:03.583990 24447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:03.584071 24447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:03.588150 24447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:03.588250 24447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:03.588330 24447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:03.588410 24447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:03.594364 24447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:03.595566 24447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:03.596643 24447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:03.607462 24453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.607476 24452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:03.607476 24455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:03.607967 24447 server_base.cc:1034] running on GCE node
I20260619 06:53:03.608454 24447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:03.609231 24447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:03.610435 24447 hybrid_clock.cc:648] HybridClock initialized: now 1781851983610420 us; error 30 us; skew 500 ppm
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:03.613456 24447 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:03.614477 24447 webserver.cc:460] Webserver started at http://127.22.113.3:41477/ using document root <none> and password file <none>
I20260619 06:53:03.615072 24447 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:03.615160 24447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:03.615445 24447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:03.617542 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "6c88d42578b44dc69420e835a705b3b2"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "8947d808f38a3d3dbcb95fd2d3da2890"
server_key_iv: "e474805d6b15e5dfaec50609c890c9cd"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.618108 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "6c88d42578b44dc69420e835a705b3b2"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "8947d808f38a3d3dbcb95fd2d3da2890"
server_key_iv: "e474805d6b15e5dfaec50609c890c9cd"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.622174 24447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:03.624966 24462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.626381 24447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:03.626550 24447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6c88d42578b44dc69420e835a705b3b2"
format_stamp: "Formatted at 2026-06-19 06:53:03 on dist-test-slave-g8zg"
server_key: "8947d808f38a3d3dbcb95fd2d3da2890"
server_key_iv: "e474805d6b15e5dfaec50609c890c9cd"
server_key_version: "encryptionkey@0"
I20260619 06:53:03.626672 24447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:03.650830 24447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:03.653973 24447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:03.654197 24447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:03.655920 24447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:03.655994 24447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:03.656065 24447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:03.656109 24447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:03.667874 24447 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:37569
I20260619 06:53:03.668231 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:37569 every 8 connection(s)
I20260619 06:53:03.669211 24447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:03.670140 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24447
I20260619 06:53:03.670269 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:03.670620 22980 external_mini_cluster.cc:1428] Setting key a36df222d9a01717969375f8f9f002ba
I20260619 06:53:03.671912 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.657046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:60739 (local address 127.22.113.62:42903)
0619 06:53:03.657237 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.657242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.658300 (+ 1058us) server_negotiation.cc:407] Connection header received
0619 06:53:03.659752 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.659755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.659819 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.659912 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.661634 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.662271 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.663761 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.664213 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.664396 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.667471 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.667478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.667484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.667523 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.669303 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.669957 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.669961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.669964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.670065 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.670561 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.670564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.670565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.670826 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.670911 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.671589 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.671760 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:53:03.672907 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.657571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.3:60739)
0619 06:53:03.658099 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.658155 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.659495 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.660213 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.660219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.660764 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.661494 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.661508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.662395 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.662398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.663601 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.663611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.664451 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.664456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.664651 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.665302 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.665319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.667212 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.669460 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.669464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.669475 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.669827 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.670205 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.670210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.670213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.670431 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.670942 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.670949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.671345 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.671520 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.671920 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":163,"threads_started":1}
I20260619 06:53:03.674347 24578 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42903
I20260619 06:53:03.674535 24578 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:03.675033 24578 heartbeater.cc:511] Master 127.22.113.62:42903 requested a full tablet report, sending...
I20260619 06:53:03.676151 24095 ts_manager.cc:194] Registered new tserver with Master: 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3:37569)
I20260619 06:53:03.676750 24095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:60739
I20260619 06:53:03.684347 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.676849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.22.113.3:38099)
0619 06:53:03.677165 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.677178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.677297 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.677540 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.677543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.677774 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:03.677916 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.677922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.678731 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.678735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.679398 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.679407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.679881 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.679883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.679951 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.680458 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:03.680468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:03.680767 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:03.682661 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.682663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.682665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.682835 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.683152 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:03.683155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:03.683157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:03.683293 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:03.683760 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:03.683765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:03.683860 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.683972 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.684130 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260619 06:53:03.684726 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.676891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:38099 (local address 127.22.113.62:42903)
0619 06:53:03.677017 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.677021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.677255 (+ 234us) server_negotiation.cc:407] Connection header received
0619 06:53:03.677389 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.677392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.677434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.677524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.678023 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.678585 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.679522 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.679764 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.679868 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.680942 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.680945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.680947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.680971 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.682561 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.682943 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.682946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.682947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.683031 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.683384 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.683387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.683389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.683620 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.683742 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.684058 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.684210 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:53:03.685231 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:53:03 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:03.714157 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.705509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.22.113.62:42903)
0619 06:53:03.705660 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.705664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.705676 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:03.705729 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.705731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.705777 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.705877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:03.706616 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.707198 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.708239 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.708448 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.708528 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.711062 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:03.711065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:03.711067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:03.711087 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:03.712585 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.713006 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.713008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.713009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.713077 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:03.713373 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:03.713375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:03.713376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:03.713536 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:03.713642 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.713919 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.714015 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:53:03.717602 24095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:03.721156 24095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:03.738073 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.730932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.22.113.3:37569)
0619 06:53:03.731464 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.731479 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.732229 (+ 750us) server_negotiation.cc:407] Connection header received
0619 06:53:03.732312 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.732317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.732547 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.732707 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:03.733310 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.734521 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.736387 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737047 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737208 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737394 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.737771 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.737887 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":130,"threads_started":1}
I20260619 06:53:03.738282 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.731913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.22.113.1:44765)
0619 06:53:03.732340 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.732348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.732595 (+ 247us) server_negotiation.cc:407] Connection header received
0619 06:53:03.732872 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.732877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.733146 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.733301 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:03.733763 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.734864 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.736542 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737406 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737522 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737696 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.738045 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.738139 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":100,"threads_started":1}
I20260619 06:53:03.738296 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.731366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.22.113.2:46183)
0619 06:53:03.732358 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.732369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.732386 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:03.732446 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.732450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.732587 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.732723 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:03.733179 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.734289 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.736430 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737400 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737522 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737699 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.737981 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.738102 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":108,"threads_started":1}
I20260619 06:53:03.738742 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.730792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37569 (local address 127.0.0.1:48506)
0619 06:53:03.731870 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.731932 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.732085 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.732870 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.732875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.732900 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:03.733154 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.733165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.734839 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.734843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.736234 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.736247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737206 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.737361 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737410 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.737530 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.738586 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":56,"spinlock_wait_cycles":4096,"thread_start_us":98,"threads_started":1}
I20260619 06:53:03.738909 24599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.731687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:44765 (local address 127.0.0.1:37986)
0619 06:53:03.732495 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.732509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.732711 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.733484 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.733487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.733504 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:03.733593 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.733598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.735018 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.735021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.736285 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.736296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737765 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.737866 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737886 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.738557 (+ 671us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.738655 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":105,"thread_start_us":108,"threads_started":1}
I20260619 06:53:03.738823 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.730934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:46183 (local address 127.0.0.1:47800)
0619 06:53:03.731870 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.731932 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.732085 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.732750 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.732760 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.732787 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:03.733053 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.733062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.734412 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.734416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.736315 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.736325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.737527 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.737530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.737736 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.737830 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.737918 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.738000 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":489,"spinlock_wait_cycles":5248,"thread_start_us":84,"threads_started":1}
I20260619 06:53:03.742038 24372 tablet_service.cc:1467] Processing CreateTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 (DEFAULT_TABLE table=test-table [id=cec0999b30674da4b6516c2f7d112b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.742569 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 (DEFAULT_TABLE table=test-table [id=cec0999b30674da4b6516c2f7d112b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.742566 24232 tablet_service.cc:1467] Processing CreateTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 (DEFAULT_TABLE table=test-table [id=cec0999b30674da4b6516c2f7d112b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.743281 24372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d62b8afa23b4561be116f3e0961fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.743779 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d62b8afa23b4561be116f3e0961fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.743779 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d62b8afa23b4561be116f3e0961fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.752839 24601 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Bootstrap starting.
I20260619 06:53:03.752893 24602 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Bootstrap starting.
I20260619 06:53:03.753356 24603 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Bootstrap starting.
I20260619 06:53:03.755668 24603 tablet_bootstrap.cc:654] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.755668 24602 tablet_bootstrap.cc:654] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.756672 24602 log.cc:824] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:03.756723 24601 tablet_bootstrap.cc:654] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.757630 24603 log.cc:824] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:03.758176 24601 log.cc:824] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:03.759086 24602 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: No bootstrap required, opened a new log
I20260619 06:53:03.759312 24603 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: No bootstrap required, opened a new log
I20260619 06:53:03.759336 24602 ts_tablet_manager.cc:1397] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:53:03.759470 24603 ts_tablet_manager.cc:1397] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:53:03.761229 24601 tablet_bootstrap.cc:492] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: No bootstrap required, opened a new log
I20260619 06:53:03.761489 24601 ts_tablet_manager.cc:1397] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:53:03.764122 24602 raft_consensus.cc:348] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.764281 24603 raft_consensus.cc:348] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.764479 24603 raft_consensus.cc:374] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.764472 24602 raft_consensus.cc:374] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.764535 24602 raft_consensus.cc:729] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c88d42578b44dc69420e835a705b3b2, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.764535 24603 raft_consensus.cc:729] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb8d9cf4b0f4977a33eb97497b683fc, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.765072 24602 consensus_queue.cc:260] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.765072 24603 consensus_queue.cc:260] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.765105 24601 raft_consensus.cc:348] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.765422 24601 raft_consensus.cc:374] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.765502 24601 raft_consensus.cc:729] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d254570b7c043efa2869969b6bc4e77, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.765954 24601 consensus_queue.cc:260] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.766235 24603 ts_tablet_manager.cc:1428] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:53:03.766235 24602 ts_tablet_manager.cc:1428] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:53:03.766669 24298 heartbeater.cc:503] Master 127.22.113.62:42903 was elected leader, sending a full tablet report...
I20260619 06:53:03.766712 24438 heartbeater.cc:503] Master 127.22.113.62:42903 was elected leader, sending a full tablet report...
I20260619 06:53:03.767066 24601 ts_tablet_manager.cc:1428] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:53:03.767588 24578 heartbeater.cc:503] Master 127.22.113.62:42903 was elected leader, sending a full tablet report...
I20260619 06:53:03.864751 24608 raft_consensus.cc:482] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:03.865006 24608 raft_consensus.cc:504] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.866407 24608 leader_election.cc:288] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183), 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765)
I20260619 06:53:03.870973 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.866652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:46183 (local address 127.22.113.3:34739)
0619 06:53:03.866822 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.866841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.866948 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.867328 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.867338 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.867372 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:03.867604 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.867614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.868574 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.868578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.869530 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.869541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.870482 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.870485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.870619 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.870650 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.870732 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.870815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"spinlock_wait_cycles":47872}
I20260619 06:53:03.871006 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.866855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:34739 (local address 127.22.113.2:46183)
0619 06:53:03.867019 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.867025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.867042 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:03.867119 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.867123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.867178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.867291 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:03.867770 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.868434 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.869714 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.870338 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.870469 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.870523 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.870776 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.870871 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:53:03.871762 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.867077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:48617 (local address 127.22.113.1:44765)
0619 06:53:03.867276 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.867283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.867435 (+ 152us) server_negotiation.cc:407] Connection header received
0619 06:53:03.867588 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.867592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.867650 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.867748 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:03.868075 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.868693 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.870319 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.870982 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.871127 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.871193 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.871523 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.871622 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:53:03.871840 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d62b8afa23b4561be116f3e0961fa54" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77" is_pre_election: true
I20260619 06:53:03.872154 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.866854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:44765 (local address 127.22.113.3:48617)
0619 06:53:03.867327 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:03.867341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:03.867472 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:03.867770 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:03.867773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:03.867789 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:03.867929 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.867936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.868816 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.868818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.870164 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:03.870174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.871282 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.871286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:03.871379 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.871400 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:53:03.871923 (+ 523us) client_negotiation.cc:240] Negotiation successful
0619 06:53:03.872005 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"spinlock_wait_cycles":2304,"thread_start_us":168,"threads_started":1}
I20260619 06:53:03.872205 24392 raft_consensus.cc:2454] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 0.
I20260619 06:53:03.872776 24465 leader_election.cc:302] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c88d42578b44dc69420e835a705b3b2, 6d254570b7c043efa2869969b6bc4e77; no voters:
I20260619 06:53:03.872993 24252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d62b8afa23b4561be116f3e0961fa54" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" is_pre_election: true
I20260619 06:53:03.873134 24608 raft_consensus.cc:2781] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:03.873261 24608 raft_consensus.cc:482] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:03.873313 24608 raft_consensus.cc:3037] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:03.873296 24252 raft_consensus.cc:2454] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 0.
I20260619 06:53:03.874817 24608 raft_consensus.cc:504] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.875314 24608 leader_election.cc:288] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183), 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765)
I20260619 06:53:03.875780 24252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d62b8afa23b4561be116f3e0961fa54" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc"
I20260619 06:53:03.875963 24252 raft_consensus.cc:3037] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:03.877295 24252 raft_consensus.cc:2454] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 1.
I20260619 06:53:03.877636 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d62b8afa23b4561be116f3e0961fa54" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77"
I20260619 06:53:03.877663 24466 leader_election.cc:302] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c88d42578b44dc69420e835a705b3b2, 9bb8d9cf4b0f4977a33eb97497b683fc; no voters:
I20260619 06:53:03.877786 24392 raft_consensus.cc:3037] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:03.879372 24608 raft_consensus.cc:2781] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:03.879397 24392 raft_consensus.cc:2454] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 1.
I20260619 06:53:03.879683 24608 raft_consensus.cc:686] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6c88d42578b44dc69420e835a705b3b2, State: Running, Role: LEADER
I20260619 06:53:03.880082 24608 consensus_queue.cc:237] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } }
I20260619 06:53:03.883800 24095 catalog_manager.cc:5495] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "6c88d42578b44dc69420e835a705b3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:03.908931 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:03.905131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.22.113.3:37569)
0619 06:53:03.905329 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:03.905334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:03.905351 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:03.905505 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:03.905508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:03.905564 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:03.905669 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:03.905935 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.906457 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.907388 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:03.907623 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:03.907737 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:03.908069 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:03.908495 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:03.908618 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:03.908696 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 06:53:03.908754 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 06:53:03.915397 24392 raft_consensus.cc:1261] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Refusing update from remote peer 6c88d42578b44dc69420e835a705b3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:03.915477 24252 raft_consensus.cc:1261] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Refusing update from remote peer 6c88d42578b44dc69420e835a705b3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:03.916086 24608 consensus_queue.cc:1035] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:03.916289 24611 consensus_queue.cc:1035] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:03.921423 24579 tablet.cc:2369] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:03.940431 24095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:03.940675 24095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:03.943320 24095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Sending DeleteTablet for 3 replicas of tablet 0d62b8afa23b4561be116f3e0961fa54
I20260619 06:53:03.944363 24232 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:03 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37986
I20260619 06:53:03.944399 24372 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:03 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:47800
I20260619 06:53:03.944964 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d62b8afa23b4561be116f3e0961fa54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:03 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:48506
I20260619 06:53:03.945207 24624 tablet_replica.cc:330] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: stopping tablet replica
I20260619 06:53:03.945560 24625 tablet_replica.cc:330] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: stopping tablet replica
I20260619 06:53:03.945650 24624 raft_consensus.cc:2229] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:03.946007 24625 raft_consensus.cc:2229] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:03.945924 24095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:03.946419 24095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:03.946440 24624 raft_consensus.cc:2258] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:03.946502 24625 raft_consensus.cc:2258] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:03.948431 24624 ts_tablet_manager.cc:1905] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:03.949472 24625 ts_tablet_manager.cc:1905] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:03.950098 24626 tablet_replica.cc:330] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: stopping tablet replica
I20260619 06:53:03.950476 24626 raft_consensus.cc:2229] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:03.950971 24626 raft_consensus.cc:2258] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:03.952091 24624 ts_tablet_manager.cc:1918] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:03.952229 24624 log.cc:1196] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/0d62b8afa23b4561be116f3e0961fa54
I20260619 06:53:03.952584 24624 ts_tablet_manager.cc:1939] T 0d62b8afa23b4561be116f3e0961fa54 P 6d254570b7c043efa2869969b6bc4e77: Deleting consensus metadata
I20260619 06:53:03.952732 24625 ts_tablet_manager.cc:1918] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:03.952834 24625 log.cc:1196] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/0d62b8afa23b4561be116f3e0961fa54
I20260619 06:53:03.952867 24372 tablet_service.cc:1467] Processing CreateTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 (DEFAULT_TABLE table=test-table [id=a59ee697c6a14f2f8816990173f3498d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.953172 24372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195fe7a1faf4ec7ad933fc3df946115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.953231 24625 ts_tablet_manager.cc:1939] T 0d62b8afa23b4561be116f3e0961fa54 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting consensus metadata
I20260619 06:53:03.953385 24232 tablet_service.cc:1467] Processing CreateTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 (DEFAULT_TABLE table=test-table [id=a59ee697c6a14f2f8816990173f3498d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.953663 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195fe7a1faf4ec7ad933fc3df946115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.953764 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 (DEFAULT_TABLE table=test-table [id=a59ee697c6a14f2f8816990173f3498d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:03.954023 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8195fe7a1faf4ec7ad933fc3df946115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:03.955152 24601 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Bootstrap starting.
I20260619 06:53:03.956256 24601 tablet_bootstrap.cc:654] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.956553 24602 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Bootstrap starting.
I20260619 06:53:03.956965 24603 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Bootstrap starting.
I20260619 06:53:03.957199 24081 catalog_manager.cc:4841] TS 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183): tablet 0d62b8afa23b4561be116f3e0961fa54 (table test-table [id=cec0999b30674da4b6516c2f7d112b56]) successfully deleted
I20260619 06:53:03.957639 24602 tablet_bootstrap.cc:654] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.957849 24626 ts_tablet_manager.cc:1905] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:03.958027 24603 tablet_bootstrap.cc:654] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:03.958338 24082 catalog_manager.cc:4841] TS 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765): tablet 0d62b8afa23b4561be116f3e0961fa54 (table test-table [id=cec0999b30674da4b6516c2f7d112b56]) successfully deleted
I20260619 06:53:03.959311 24602 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: No bootstrap required, opened a new log
I20260619 06:53:03.959419 24602 ts_tablet_manager.cc:1397] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:03.959430 24603 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: No bootstrap required, opened a new log
I20260619 06:53:03.959512 24603 ts_tablet_manager.cc:1397] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:03.959924 24602 raft_consensus.cc:348] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.960053 24602 raft_consensus.cc:374] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.960103 24602 raft_consensus.cc:729] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c88d42578b44dc69420e835a705b3b2, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.960013 24603 raft_consensus.cc:348] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.960162 24603 raft_consensus.cc:374] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.960218 24603 raft_consensus.cc:729] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb8d9cf4b0f4977a33eb97497b683fc, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.960240 24602 consensus_queue.cc:260] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.960357 24603 consensus_queue.cc:260] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.960510 24602 ts_tablet_manager.cc:1428] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:03.960644 24603 ts_tablet_manager.cc:1428] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:03.962903 24601 tablet_bootstrap.cc:492] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: No bootstrap required, opened a new log
I20260619 06:53:03.963021 24601 ts_tablet_manager.cc:1397] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260619 06:53:03.963495 24626 ts_tablet_manager.cc:1918] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:03.963629 24626 log.cc:1196] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/0d62b8afa23b4561be116f3e0961fa54
I20260619 06:53:03.963579 24601 raft_consensus.cc:348] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.963713 24601 raft_consensus.cc:374] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:03.963768 24601 raft_consensus.cc:729] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d254570b7c043efa2869969b6bc4e77, State: Initialized, Role: FOLLOWER
I20260619 06:53:03.963922 24601 consensus_queue.cc:260] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:03.964044 24626 ts_tablet_manager.cc:1939] T 0d62b8afa23b4561be116f3e0961fa54 P 6c88d42578b44dc69420e835a705b3b2: Deleting consensus metadata
I20260619 06:53:03.964191 24601 ts_tablet_manager.cc:1428] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:03.965562 24082 catalog_manager.cc:4841] TS 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3:37569): tablet 0d62b8afa23b4561be116f3e0961fa54 (table test-table [id=cec0999b30674da4b6516c2f7d112b56]) successfully deleted
I20260619 06:53:04.046250 24608 raft_consensus.cc:482] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:04.046487 24608 raft_consensus.cc:504] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.047158 24608 leader_election.cc:288] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765), 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183)
I20260619 06:53:04.047794 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195fe7a1faf4ec7ad933fc3df946115" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77" is_pre_election: true
I20260619 06:53:04.047823 24252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195fe7a1faf4ec7ad933fc3df946115" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" is_pre_election: true
I20260619 06:53:04.047976 24252 raft_consensus.cc:2454] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 0.
I20260619 06:53:04.047976 24392 raft_consensus.cc:2454] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 0.
I20260619 06:53:04.048301 24466 leader_election.cc:302] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c88d42578b44dc69420e835a705b3b2, 9bb8d9cf4b0f4977a33eb97497b683fc; no voters:
I20260619 06:53:04.048492 24608 raft_consensus.cc:2781] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:04.048549 24608 raft_consensus.cc:482] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:04.048589 24608 raft_consensus.cc:3037] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.049795 24608 raft_consensus.cc:504] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.050145 24608 leader_election.cc:288] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765), 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183)
I20260619 06:53:04.050612 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195fe7a1faf4ec7ad933fc3df946115" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77"
I20260619 06:53:04.050637 24252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8195fe7a1faf4ec7ad933fc3df946115" candidate_uuid: "6c88d42578b44dc69420e835a705b3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc"
I20260619 06:53:04.050763 24252 raft_consensus.cc:3037] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.050765 24392 raft_consensus.cc:3037] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.052217 24392 raft_consensus.cc:2454] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 1.
I20260619 06:53:04.052764 24252 raft_consensus.cc:2454] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c88d42578b44dc69420e835a705b3b2 in term 1.
I20260619 06:53:04.053177 24465 leader_election.cc:302] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c88d42578b44dc69420e835a705b3b2, 6d254570b7c043efa2869969b6bc4e77; no voters:
I20260619 06:53:04.053383 24608 raft_consensus.cc:2781] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:04.053496 24608 raft_consensus.cc:686] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6c88d42578b44dc69420e835a705b3b2, State: Running, Role: LEADER
I20260619 06:53:04.053882 24608 consensus_queue.cc:237] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.056296 24095 catalog_manager.cc:5495] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "6c88d42578b44dc69420e835a705b3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } health_report { overall_health: HEALTHY } } }
Jun 19 06:53:04 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:04.112108 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.101778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42903 (local address 127.0.0.1:60092)
0619 06:53:04.102130 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.102149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.102288 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.102645 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.102647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.103362 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:04.103485 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.103492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.104579 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.104581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.105191 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.105199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.105802 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.105805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.105876 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.106565 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:04.106578 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:04.108407 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:04.110209 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:04.110222 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:04.110227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:04.110579 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:04.110879 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:04.110882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:04.110883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:04.111014 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:04.111529 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:04.111536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:04.111673 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.111785 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.111954 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":159,"threads_started":1}
I20260619 06:53:04.112238 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.101905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.22.113.62:42903)
0619 06:53:04.102087 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.102092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.102253 (+ 161us) server_negotiation.cc:407] Connection header received
0619 06:53:04.102407 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.102411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.102476 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.102609 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:04.103644 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.104439 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.105341 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.105633 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.105744 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.108570 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:04.108574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:04.108576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:04.108597 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:04.110075 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.110686 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.110689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.110690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.110760 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.111110 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.111112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.111114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.111403 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:04.111488 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.111847 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.112034 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":64}
I20260619 06:53:04.117621 24095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60092:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:53:04.118595 24095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:04.124540 24232 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07c6667e414a72bcd9977289b1256f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e56287cafd940f7b84e461bd3f70f2f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:04.124615 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07c6667e414a72bcd9977289b1256f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e56287cafd940f7b84e461bd3f70f2f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:04.124609 24372 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07c6667e414a72bcd9977289b1256f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e56287cafd940f7b84e461bd3f70f2f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:04.125057 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07c6667e414a72bcd9977289b1256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:04.125057 24372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07c6667e414a72bcd9977289b1256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:04.125079 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07c6667e414a72bcd9977289b1256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:04.128247 24602 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: Bootstrap starting.
I20260619 06:53:04.128377 24603 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: Bootstrap starting.
I20260619 06:53:04.129197 24602 tablet_bootstrap.cc:654] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:04.129366 24603 tablet_bootstrap.cc:654] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:04.129873 24601 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: Bootstrap starting.
I20260619 06:53:04.130764 24602 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: No bootstrap required, opened a new log
I20260619 06:53:04.130903 24602 ts_tablet_manager.cc:1397] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:04.131037 24601 tablet_bootstrap.cc:654] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:04.131558 24602 raft_consensus.cc:348] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.131737 24602 raft_consensus.cc:374] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:04.131781 24602 raft_consensus.cc:729] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c88d42578b44dc69420e835a705b3b2, State: Initialized, Role: FOLLOWER
I20260619 06:53:04.131942 24602 consensus_queue.cc:260] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.132386 24601 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: No bootstrap required, opened a new log
I20260619 06:53:04.132496 24601 ts_tablet_manager.cc:1397] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:53:04.132529 24603 tablet_bootstrap.cc:492] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: No bootstrap required, opened a new log
I20260619 06:53:04.132615 24603 ts_tablet_manager.cc:1397] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:53:04.133051 24601 raft_consensus.cc:348] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.133188 24601 raft_consensus.cc:374] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:04.133232 24601 raft_consensus.cc:729] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d254570b7c043efa2869969b6bc4e77, State: Initialized, Role: FOLLOWER
I20260619 06:53:04.133219 24603 raft_consensus.cc:348] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.133342 24603 raft_consensus.cc:374] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:04.133386 24603 raft_consensus.cc:729] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb8d9cf4b0f4977a33eb97497b683fc, State: Initialized, Role: FOLLOWER
I20260619 06:53:04.133405 24601 consensus_queue.cc:260] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.133561 24603 consensus_queue.cc:260] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.133663 24609 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.133801 24609 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.133903 24608 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.134063 24608 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.134492 24601 ts_tablet_manager.cc:1428] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:53:04.135731 24603 ts_tablet_manager.cc:1428] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:04.135648 24607 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.135808 24607 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.135897 24602 ts_tablet_manager.cc:1428] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20260619 06:53:04.179965 24439 tablet.cc:2369] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:04.188755 24299 tablet.cc:2369] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:04.307693 24607 raft_consensus.cc:482] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:04.307907 24607 raft_consensus.cc:504] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.309496 24607 leader_election.cc:288] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183), 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3:37569)
I20260619 06:53:04.314136 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.309872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37569 (local address 127.22.113.1:40287)
0619 06:53:04.310324 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.310341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.310490 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.310785 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.310787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.310806 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:04.310964 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.310971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.311824 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.311828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.312829 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.312838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.313607 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.313614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.313702 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.313727 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.313856 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.313946 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":130,"threads_started":1}
I20260619 06:53:04.314159 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.310025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40287 (local address 127.22.113.3:37569)
0619 06:53:04.310181 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.310186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.310468 (+ 282us) server_negotiation.cc:407] Connection header received
0619 06:53:04.310610 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.310613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.310661 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.310746 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:04.311095 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.311712 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.312968 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.313449 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.313587 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.313638 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.313914 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.314009 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:53:04.314321 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.310027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:33623 (local address 127.22.113.2:46183)
0619 06:53:04.310318 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.310323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.310629 (+ 306us) server_negotiation.cc:407] Connection header received
0619 06:53:04.310793 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.310797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.310851 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.310941 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:04.311311 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.311918 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.313120 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.313556 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.313670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.313742 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.314071 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.314197 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260619 06:53:04.314865 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.309872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:46183 (local address 127.22.113.1:33623)
0619 06:53:04.310486 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.310503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.310642 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.310990 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.310993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.311009 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:04.311157 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.311164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.312060 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.312063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.312977 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.312999 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.313736 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.313740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.313885 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.313920 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.314547 (+ 627us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.314643 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":68,"thread_start_us":147,"threads_started":1}
I20260619 06:53:04.314996 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07c6667e414a72bcd9977289b1256f" candidate_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c88d42578b44dc69420e835a705b3b2" is_pre_election: true
I20260619 06:53:04.315351 24532 raft_consensus.cc:2454] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb8d9cf4b0f4977a33eb97497b683fc in term 0.
I20260619 06:53:04.315352 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07c6667e414a72bcd9977289b1256f" candidate_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77" is_pre_election: true
I20260619 06:53:04.315500 24392 raft_consensus.cc:2454] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb8d9cf4b0f4977a33eb97497b683fc in term 0.
I20260619 06:53:04.315882 24185 leader_election.cc:302] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c88d42578b44dc69420e835a705b3b2, 6d254570b7c043efa2869969b6bc4e77, 9bb8d9cf4b0f4977a33eb97497b683fc; no voters:
I20260619 06:53:04.316203 24607 raft_consensus.cc:2781] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:04.316268 24607 raft_consensus.cc:482] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:04.316308 24607 raft_consensus.cc:3037] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.317517 24607 raft_consensus.cc:504] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.317904 24607 leader_election.cc:288] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [CANDIDATE]: Term 1 election: Requested vote from peers 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183), 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3:37569)
I20260619 06:53:04.318279 24392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07c6667e414a72bcd9977289b1256f" candidate_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d254570b7c043efa2869969b6bc4e77"
I20260619 06:53:04.318423 24392 raft_consensus.cc:3037] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.318516 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07c6667e414a72bcd9977289b1256f" candidate_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c88d42578b44dc69420e835a705b3b2"
I20260619 06:53:04.318714 24532 raft_consensus.cc:3037] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:04.319620 24392 raft_consensus.cc:2454] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb8d9cf4b0f4977a33eb97497b683fc in term 1.
I20260619 06:53:04.319914 24185 leader_election.cc:302] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d254570b7c043efa2869969b6bc4e77, 9bb8d9cf4b0f4977a33eb97497b683fc; no voters:
I20260619 06:53:04.320041 24532 raft_consensus.cc:2454] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bb8d9cf4b0f4977a33eb97497b683fc in term 1.
I20260619 06:53:04.320123 24607 raft_consensus.cc:2781] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:04.320379 24607 raft_consensus.cc:686] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 LEADER]: Becoming Leader. State: Replica: 9bb8d9cf4b0f4977a33eb97497b683fc, State: Running, Role: LEADER
I20260619 06:53:04.320736 24607 consensus_queue.cc:237] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } }
I20260619 06:53:04.322136 24642 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: TxnStatusTablet state changed. Reason: New leader 9bb8d9cf4b0f4977a33eb97497b683fc. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.322305 24642 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:04.322927 24644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:53:04.323118 24644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:53:04.323770 24094 catalog_manager.cc:5495] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1). New cstate: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:04.372717 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.368243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:44765 (local address 127.0.0.1:37996)
0619 06:53:04.368527 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.368546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.368670 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.369035 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.369038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.369045 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:04.369170 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.369176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.369947 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.369954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.370598 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.370606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.371089 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.371092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.371202 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.371284 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.372354 (+ 1070us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.372374 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.372462 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.372569 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 06:53:04.372746 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.368388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.22.113.1:44765)
0619 06:53:04.368575 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.368582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.368600 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:04.368851 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.368854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.368918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.369023 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:04.369280 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.369839 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.370734 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.370976 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.371086 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.371504 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.372199 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.372353 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.372498 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.372579 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:04.376425 24392 raft_consensus.cc:1261] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Refusing update from remote peer 9bb8d9cf4b0f4977a33eb97497b683fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:04.376601 24532 raft_consensus.cc:1261] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Refusing update from remote peer 9bb8d9cf4b0f4977a33eb97497b683fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:04.377043 24642 consensus_queue.cc:1035] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:04.377265 24607 consensus_queue.cc:1035] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:04.379393 24609 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: TxnStatusTablet state changed. Reason: New leader 9bb8d9cf4b0f4977a33eb97497b683fc. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.379537 24609 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.380234 24608 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: TxnStatusTablet state changed. Reason: New leader 9bb8d9cf4b0f4977a33eb97497b683fc. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.380381 24608 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.381394 24642 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.381506 24642 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:04.383306 24609 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.383440 24609 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6d254570b7c043efa2869969b6bc4e77: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.384135 24616 mvcc.cc:205] Tried to move back new op lower bound from 7298465727999328256 to 7298465727780417536. Current Snapshot: MvccSnapshot[applied={T|T < 7298465727999328256 or (T in {7298465727999328256})}]
I20260619 06:53:04.384908 24607 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.385015 24607 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 9bb8d9cf4b0f4977a33eb97497b683fc: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:04.386003 24608 tablet_replica.cc:439] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 } } peers { permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 } } peers { permanent_uuid: "6c88d42578b44dc69420e835a705b3b2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37569 } } }
I20260619 06:53:04.386041 24614 mvcc.cc:205] Tried to move back new op lower bound from 7298465727999328256 to 7298465727780417536. Current Snapshot: MvccSnapshot[applied={T|T < 7298465727999328256}]
I20260619 06:53:04.386116 24608 tablet_replica.cc:442] T 1f07c6667e414a72bcd9977289b1256f P 6c88d42578b44dc69420e835a705b3b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:04.399489 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.395900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:44765 (local address 127.22.113.3:45187)
0619 06:53:04.396252 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.396265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.396409 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.396664 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.396667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.396675 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:04.396798 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.396803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.397550 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.397554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.398195 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.398203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.398645 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.398649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.398731 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.398817 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.399149 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.399158 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.399235 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.399350 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260619 06:53:04.399489 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.395978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:45187 (local address 127.22.113.1:44765)
0619 06:53:04.396106 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.396110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.396348 (+ 238us) server_negotiation.cc:407] Connection header received
0619 06:53:04.396511 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.396514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.396567 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.396641 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:04.396896 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.397435 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.398314 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.398536 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.398642 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.398988 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.399065 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.399144 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.399278 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.399328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:53:04.407832 24392 raft_consensus.cc:1261] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Refusing update from remote peer 6c88d42578b44dc69420e835a705b3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:04.408000 24252 raft_consensus.cc:1261] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Refusing update from remote peer 6c88d42578b44dc69420e835a705b3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:04.408278 24611 consensus_queue.cc:1035] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d254570b7c043efa2869969b6bc4e77" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:04.408524 24608 consensus_queue.cc:1035] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb8d9cf4b0f4977a33eb97497b683fc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:04.417212 24615 mvcc.cc:205] Tried to move back new op lower bound from 7298465728130621440 to 7298465726686117888. Current Snapshot: MvccSnapshot[applied={T|T < 7298465728130621440}]
I20260619 06:53:04.441788 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.433573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:42027 (local address 127.22.113.3:37569)
0619 06:53:04.433801 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.433807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.437074 (+ 3267us) server_negotiation.cc:407] Connection header received
0619 06:53:04.437337 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.437342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.437415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.437511 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:04.437871 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.438697 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.440249 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.440638 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.440755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.441148 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.441267 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.441345 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.441527 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.441613 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 06:53:04.442332 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.433191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37569 (local address 127.22.113.1:42027)
0619 06:53:04.436966 (+ 3775us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:04.436986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:04.437170 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:04.437552 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:04.437558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:04.437569 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:04.437717 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.437723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.438903 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.438908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.440104 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:04.440116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.440802 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.440806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:04.440917 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.440979 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:04.441371 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:04.441380 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:04.442035 (+ 655us) client_negotiation.cc:240] Negotiation successful
0619 06:53:04.442129 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":106,"threads_started":1}
I20260619 06:53:04.472450 24093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:04.472697 24093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:04.475330 24093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4887613610414116abff24f95b6a0c25: Sending DeleteTablet for 3 replicas of tablet 8195fe7a1faf4ec7ad933fc3df946115
I20260619 06:53:04.476084 24372 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:04 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:47800
I20260619 06:53:04.476140 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:04 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:48506
I20260619 06:53:04.476563 24658 tablet_replica.cc:330] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: stopping tablet replica
I20260619 06:53:04.476689 24658 raft_consensus.cc:2229] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:04.476796 24658 raft_consensus.cc:2258] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:04.477016 24659 tablet_replica.cc:330] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: stopping tablet replica
I20260619 06:53:04.477144 24659 raft_consensus.cc:2229] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:04.477324 24659 raft_consensus.cc:2258] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:04.477626 24232 tablet_service.cc:1514] Processing DeleteTablet for tablet 8195fe7a1faf4ec7ad933fc3df946115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:04 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37986
I20260619 06:53:04.478001 24658 ts_tablet_manager.cc:1905] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:04.478063 24661 tablet_replica.cc:330] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: stopping tablet replica
I20260619 06:53:04.478204 24661 raft_consensus.cc:2229] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:04.478329 24661 raft_consensus.cc:2258] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:04.479066 24661 ts_tablet_manager.cc:1905] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:04.481757 24658 ts_tablet_manager.cc:1918] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:04.481846 24658 log.cc:1196] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/8195fe7a1faf4ec7ad933fc3df946115
I20260619 06:53:04.482128 24658 ts_tablet_manager.cc:1939] T 8195fe7a1faf4ec7ad933fc3df946115 P 6d254570b7c043efa2869969b6bc4e77: Deleting consensus metadata
I20260619 06:53:04.482367 24661 ts_tablet_manager.cc:1918] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:04.482452 24661 log.cc:1196] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/8195fe7a1faf4ec7ad933fc3df946115
I20260619 06:53:04.482724 24661 ts_tablet_manager.cc:1939] T 8195fe7a1faf4ec7ad933fc3df946115 P 9bb8d9cf4b0f4977a33eb97497b683fc: Deleting consensus metadata
I20260619 06:53:04.483045 24081 catalog_manager.cc:4841] TS 6d254570b7c043efa2869969b6bc4e77 (127.22.113.2:46183): tablet 8195fe7a1faf4ec7ad933fc3df946115 (table test-table [id=a59ee697c6a14f2f8816990173f3498d]) successfully deleted
I20260619 06:53:04.483625 24082 catalog_manager.cc:4841] TS 9bb8d9cf4b0f4977a33eb97497b683fc (127.22.113.1:44765): tablet 8195fe7a1faf4ec7ad933fc3df946115 (table test-table [id=a59ee697c6a14f2f8816990173f3498d]) successfully deleted
I20260619 06:53:04.483808 24659 ts_tablet_manager.cc:1905] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:04.486810 24659 ts_tablet_manager.cc:1918] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:04.486907 24659 log.cc:1196] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/8195fe7a1faf4ec7ad933fc3df946115
I20260619 06:53:04.487231 24659 ts_tablet_manager.cc:1939] T 8195fe7a1faf4ec7ad933fc3df946115 P 6c88d42578b44dc69420e835a705b3b2: Deleting consensus metadata
I20260619 06:53:04.488077 24082 catalog_manager.cc:4841] TS 6c88d42578b44dc69420e835a705b3b2 (127.22.113.3:37569): tablet 8195fe7a1faf4ec7ad933fc3df946115 (table test-table [id=a59ee697c6a14f2f8816990173f3498d]) successfully deleted
Jun 19 06:53:04 dist-test-slave-g8zg krb5kdc[24050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:04.495353 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.478386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.22.113.1:44765)
0619 06:53:04.478559 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.478563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.483692 (+ 5129us) server_negotiation.cc:407] Connection header received
0619 06:53:04.483922 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.483925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.483985 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.484040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:04.485140 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.485946 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.487211 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.487641 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.487767 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.490679 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:04.490693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:04.490715 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:04.490763 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:04.493195 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.493716 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.493727 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.493730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.493856 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.494213 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.494217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.494218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.494565 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:04.494777 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.494994 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.495123 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 06:53:04.495898 24292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38008
I20260619 06:53:04.506839 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:04.498012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.22.113.62:42903)
0619 06:53:04.498176 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:04.498180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:04.498194 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:53:04.498340 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:04.498343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:04.498396 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:04.498471 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:04.499366 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.500062 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.501020 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:04.501443 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:04.501604 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:04.503344 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:04.503348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:04.503352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:04.503384 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:04.505122 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.505576 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.505579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.505581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.505675 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:04.505942 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:04.505945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:04.505947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:04.506187 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:04.506339 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:04.506549 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 06:53:04.506673 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:53:04.507493 24093 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60102
I20260619 06:53:04.508828 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24167
I20260619 06:53:04.517089 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24307
I20260619 06:53:04.524055 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24447
I20260619 06:53:04.531657 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24063
2026-06-19T06:53:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3073 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:04 dist-test-slave-g8zg krb5kdc[24667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:04 dist-test-slave-g8zg krb5kdc[24667](info): set up 2 sockets
Jun 19 06:53:04 dist-test-slave-g8zg krb5kdc[24667](info): commencing operation
krb5kdc: starting...
W20260619 06:53:05.627290 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:05 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:05.770895 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42579
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:42579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:05.903781 24680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:05.904129 24680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:05.904201 24680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:05.908452 24680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:05.908571 24680 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:53:05.908607 24680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:05.908653 24680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:05.908713 24680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:05.908753 24680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:05.914470 24680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:42579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:05.915778 24680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:05.916983 24680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:05.927057 24685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:05.927426 24680 server_base.cc:1034] running on GCE node
W20260619 06:53:05.927063 24686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:05.927060 24688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:05.928345 24680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:05.929667 24680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:05.930867 24680 hybrid_clock.cc:648] HybridClock initialized: now 1781851985930828 us; error 57 us; skew 500 ppm
Jun 19 06:53:05 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851985, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:05.934070 24680 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:05.935089 24680 webserver.cc:460] Webserver started at http://127.22.113.62:36085/ using document root <none> and password file <none>
I20260619 06:53:05.935756 24680 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:05.935832 24680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:05.936019 24680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:05.938169 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "a9a9e6e7db8e4af29e56974e6ff6175b"
format_stamp: "Formatted at 2026-06-19 06:53:05 on dist-test-slave-g8zg"
server_key: "b0946f860e4c0f67bdc17e95c0aaa946"
server_key_iv: "bb6e7ff414d732f738c4b191c76cd6cf"
server_key_version: "encryptionkey@0"
I20260619 06:53:05.938685 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "a9a9e6e7db8e4af29e56974e6ff6175b"
format_stamp: "Formatted at 2026-06-19 06:53:05 on dist-test-slave-g8zg"
server_key: "b0946f860e4c0f67bdc17e95c0aaa946"
server_key_iv: "bb6e7ff414d732f738c4b191c76cd6cf"
server_key_version: "encryptionkey@0"
I20260619 06:53:05.942792 24680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:05.945737 24695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:05.947209 24680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:05.947427 24680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "a9a9e6e7db8e4af29e56974e6ff6175b"
format_stamp: "Formatted at 2026-06-19 06:53:05 on dist-test-slave-g8zg"
server_key: "b0946f860e4c0f67bdc17e95c0aaa946"
server_key_iv: "bb6e7ff414d732f738c4b191c76cd6cf"
server_key_version: "encryptionkey@0"
I20260619 06:53:05.947557 24680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:05.961793 24680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:05.964900 24680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:05.965116 24680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:05.974404 24680 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:42579
I20260619 06:53:05.974488 24757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:42579 every 8 connection(s)
I20260619 06:53:05.975611 24680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:05.979511 24758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:05.979918 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24680
I20260619 06:53:05.980059 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:05.980333 22980 external_mini_cluster.cc:1428] Setting key 9abe45ac2466254d97eb54bfea80836c
I20260619 06:53:05.988422 24758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Bootstrap starting.
Jun 19 06:53:05 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:05.991653 24758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:05.992766 24758 log.cc:824] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:05.995908 24758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: No bootstrap required, opened a new log
I20260619 06:53:05.998786 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:05.982113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.22.113.62:42579)
0619 06:53:05.982690 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:05.982715 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:05.982772 (+ 57us) server_negotiation.cc:407] Connection header received
0619 06:53:05.983735 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:05.983765 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:05.984296 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:05.984781 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:05.985592 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:05.987194 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:05.988258 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:05.988976 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:05.989152 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:05.991989 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:05.991998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:05.992023 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:05.992061 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:05.994882 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:05.995514 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:05.995520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:05.995527 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:05.995699 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:05.996100 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:05.996103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:05.996105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:05.996737 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:05.996948 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:05.997273 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 06:53:05.997618 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":156,"threads_started":1}
I20260619 06:53:06.000527 24758 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } }
I20260619 06:53:06.000826 24758 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:06.000901 24758 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a9e6e7db8e4af29e56974e6ff6175b, State: Initialized, Role: FOLLOWER
I20260619 06:53:06.001458 24758 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } }
I20260619 06:53:06.001670 24758 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:06.001786 24758 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:06.001901 24758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:06.003118 24758 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } }
I20260619 06:53:06.003583 24758 leader_election.cc:302] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a9e6e7db8e4af29e56974e6ff6175b; no voters:
I20260619 06:53:06.003960 24758 leader_election.cc:288] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:06.004204 24763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:06.004655 24763 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [term 1 LEADER]: Becoming Leader. State: Replica: a9a9e6e7db8e4af29e56974e6ff6175b, State: Running, Role: LEADER
I20260619 06:53:06.005067 24763 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } }
I20260619 06:53:06.005661 24758 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:06.007305 24765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a9e6e7db8e4af29e56974e6ff6175b. Latest consensus state: current_term: 1 leader_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } } }
I20260619 06:53:06.007457 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a9e6e7db8e4af29e56974e6ff6175b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42579 } } }
I20260619 06:53:06.007475 24765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:06.007655 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:06.008253 24773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:06.011654 24773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:06.018047 24773 catalog_manager.cc:1347] Generated new cluster ID: 66d48181f2f244b1817a5d53ccc23711
I20260619 06:53:06.018152 24773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:06.025399 24773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:06.026500 24773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:06.034353 24773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Generated new TSK 0
I20260619 06:53:06.035143 24773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:06.074734 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:06.197348 24784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:06.197652 24784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:06.197707 24784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:06.202015 24784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:06.202090 24784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:06.202134 24784 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:53:06.202150 24784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:06.202219 24784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:06.208261 24784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:06.209467 24784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:06.210611 24784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:06.221560 24792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:06.221534 24789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:06.221503 24784 server_base.cc:1034] running on GCE node
W20260619 06:53:06.221498 24790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:06.222256 24784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:06.222930 24784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:06.224124 24784 hybrid_clock.cc:648] HybridClock initialized: now 1781851986224091 us; error 53 us; skew 500 ppm
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:06.227233 24784 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:06.228281 24784 webserver.cc:460] Webserver started at http://127.22.113.1:37047/ using document root <none> and password file <none>
I20260619 06:53:06.228878 24784 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:06.228981 24784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:06.229223 24784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:06.231431 24784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "e1864da7500a4c6b833b5c99561709be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "6a96c06d8bdc79ca37ddc9d84d09ed6e"
server_key_iv: "5392665e5b8e814ad11353c21983257a"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.231997 24784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "e1864da7500a4c6b833b5c99561709be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "6a96c06d8bdc79ca37ddc9d84d09ed6e"
server_key_iv: "5392665e5b8e814ad11353c21983257a"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.236198 24784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:06.239354 24799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.240945 24784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:53:06.241159 24784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "e1864da7500a4c6b833b5c99561709be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "6a96c06d8bdc79ca37ddc9d84d09ed6e"
server_key_iv: "5392665e5b8e814ad11353c21983257a"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.241295 24784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:06.267143 24784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:06.270586 24784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:06.270803 24784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:06.272442 24784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:06.272523 24784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.272584 24784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:06.272639 24784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:06.284318 24784 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:33561
I20260619 06:53:06.284444 24914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:33561 every 8 connection(s)
I20260619 06:53:06.285651 24784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:06.288877 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.273958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:57895 (local address 127.22.113.62:42579)
0619 06:53:06.274162 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.274168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.275310 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 06:53:06.276868 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.276873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.276961 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.277106 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.278547 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.279537 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.280885 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.281174 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.281314 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.284733 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.284738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.284741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.284781 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.286699 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.287323 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.287326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.287328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.287429 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.287767 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.287769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.287770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.287991 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.288077 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.288618 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.288735 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:53:06.289903 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.274416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.1:57895)
0619 06:53:06.275092 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.275160 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.276579 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.277327 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.277335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.277761 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.278357 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.278374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.279694 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.279698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.280736 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.280749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.281372 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.281376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.281577 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.282408 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.282443 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.284478 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.286866 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.286871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.286884 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.287161 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.287524 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.287527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.287528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.287675 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.288092 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.288097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.288391 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.288569 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.288866 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":207,"threads_started":1}
I20260619 06:53:06.291683 24915 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42579
I20260619 06:53:06.291941 24915 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:06.292069 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24784
I20260619 06:53:06.292189 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:06.292588 22980 external_mini_cluster.cc:1428] Setting key 40bcea47a1f653e01df7e3f26723c744
I20260619 06:53:06.292716 24915 heartbeater.cc:511] Master 127.22.113.62:42579 requested a full tablet report, sending...
I20260619 06:53:06.294757 24710 ts_manager.cc:194] Registered new tserver with Master: e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561)
I20260619 06:53:06.296237 24710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:57895
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:06.304407 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.294531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.1:58681)
0619 06:53:06.294922 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.294941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.295047 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.296356 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.296362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.296617 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.296759 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.296767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.297867 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.297870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.298654 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.298665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.299207 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.299211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.299353 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.299996 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.300010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.300403 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.302601 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.302606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.302608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.302831 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.303186 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.303191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.303193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.303340 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.303767 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.303770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.303896 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.304002 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.304178 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":74,"thread_start_us":155,"threads_started":1}
I20260619 06:53:06.304400 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.295386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:58681 (local address 127.22.113.62:42579)
0619 06:53:06.296066 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.296074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.296092 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:06.296140 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.296145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.296199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.296301 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.296906 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.297685 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.298807 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.299073 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.299198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.300567 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.300571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.300574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.300600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.302471 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.302939 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.302945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.302950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.303088 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.303446 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.303451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.303452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.303660 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.303769 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.304076 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.304211 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:06.328066 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:06.441812 24924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:06.442142 24924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:06.442224 24924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:06.447872 24924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:06.447950 24924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:06.447994 24924 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:53:06.448010 24924 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:06.448045 24924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:06.453490 24924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:06.454538 24924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:06.455812 24924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:06.466277 24929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:06.466277 24930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:06.466291 24932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:06.466869 24924 server_base.cc:1034] running on GCE node
I20260619 06:53:06.467415 24924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:06.468169 24924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:06.469396 24924 hybrid_clock.cc:648] HybridClock initialized: now 1781851986469370 us; error 42 us; skew 500 ppm
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:06.472504 24924 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:06.473497 24924 webserver.cc:460] Webserver started at http://127.22.113.2:46819/ using document root <none> and password file <none>
I20260619 06:53:06.474089 24924 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:06.474179 24924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:06.474427 24924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:06.476616 24924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "8d3a04d80eb14efdb8be4de61c2236b8"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "4b000ef55114598825cc6c3760f59d71"
server_key_iv: "a6a49fd2e098fd850f1dffd9bfa8a4c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.477208 24924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "8d3a04d80eb14efdb8be4de61c2236b8"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "4b000ef55114598825cc6c3760f59d71"
server_key_iv: "a6a49fd2e098fd850f1dffd9bfa8a4c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.481181 24924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:53:06.483850 24939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.485265 24924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:06.485421 24924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "8d3a04d80eb14efdb8be4de61c2236b8"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "4b000ef55114598825cc6c3760f59d71"
server_key_iv: "a6a49fd2e098fd850f1dffd9bfa8a4c4"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.485546 24924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:06.503741 24924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:06.507172 24924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:06.507400 24924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:06.509021 24924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:06.509094 24924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.509166 24924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:06.509209 24924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.520324 24924 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:37043
I20260619 06:53:06.520412 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:37043 every 8 connection(s)
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:06.521260 24924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:06.525437 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.510436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:50825 (local address 127.22.113.62:42579)
0619 06:53:06.510622 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.510628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.511788 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 06:53:06.513334 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.513341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.513423 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.513547 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.515358 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.516168 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.517651 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.517882 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.518001 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.521132 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.521136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.521139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.521166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.523057 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.523757 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.523760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.523761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.523840 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.524236 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.524239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.524241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.524458 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.524554 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.525184 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.525296 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:06.525856 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 24924
I20260619 06:53:06.525983 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:06.526309 22980 external_mini_cluster.cc:1428] Setting key 612a24df7b3e73a20fe6461d4adfb75b
I20260619 06:53:06.526396 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.510927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.2:50825)
0619 06:53:06.511597 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.511663 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.513061 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.513883 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.513894 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.514413 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.515159 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.515180 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.516378 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.516385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.517516 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.517529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.518051 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.518056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.518254 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.519026 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.519058 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.520946 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.523188 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.523193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.523204 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.523630 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.523932 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.523934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.523935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.524072 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.524606 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.524612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.524885 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.525077 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.525347 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":171,"threads_started":1}
I20260619 06:53:06.527804 25055 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42579
I20260619 06:53:06.527966 25055 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:06.528684 25055 heartbeater.cc:511] Master 127.22.113.62:42579 requested a full tablet report, sending...
I20260619 06:53:06.529939 24712 ts_manager.cc:194] Registered new tserver with Master: 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043)
I20260619 06:53:06.530732 24712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:50825
I20260619 06:53:06.539078 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.530516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.2:53187)
0619 06:53:06.530945 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.530964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.531091 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.531411 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.531414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.531647 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.531769 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.531776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.532528 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.532531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.533435 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.533448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.533883 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.533889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.534017 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.534653 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.534672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.535151 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.537441 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.537444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.537446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.537675 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.538003 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.538005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.538006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.538078 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.538620 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.538622 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.538712 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.538797 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.538933 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":100,"threads_started":1}
I20260619 06:53:06.539270 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.530606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:53187 (local address 127.22.113.62:42579)
0619 06:53:06.530751 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.530757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.531040 (+ 283us) server_negotiation.cc:407] Connection header received
0619 06:53:06.531205 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.531208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.531294 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.531392 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.531898 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.532393 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.533558 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.533763 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.533844 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.535350 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.535353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.535356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.535383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.537308 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.537793 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.537797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.537799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.537899 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.538183 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.538187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.538189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.538511 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.538644 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.538894 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.539035 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:06.562489 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:06.685276 25064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:06.685554 25064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:06.685609 25064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:06.689716 25064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:06.689779 25064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:06.689821 25064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:53:06.689836 25064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:06.689868 25064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:06.695227 25064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:42579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:06.696269 25064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:06.697273 25064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:06.707448 25070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:06.707615 25072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:06.707468 25069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:06.707891 25064 server_base.cc:1034] running on GCE node
I20260619 06:53:06.708411 25064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:06.709272 25064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:06.710481 25064 hybrid_clock.cc:648] HybridClock initialized: now 1781851986710465 us; error 47 us; skew 500 ppm
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:06.713747 25064 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:06.714782 25064 webserver.cc:460] Webserver started at http://127.22.113.3:33587/ using document root <none> and password file <none>
I20260619 06:53:06.715579 25064 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:06.715678 25064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:06.715938 25064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:06.718334 25064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "3162ac0d7b6945b69080a39f08ad88be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "e9b782b765b2a4f82e45e94c32bbf5b7"
server_key_iv: "5e059fd082246d257f65d868ac8f549b"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.718951 25064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "3162ac0d7b6945b69080a39f08ad88be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "e9b782b765b2a4f82e45e94c32bbf5b7"
server_key_iv: "5e059fd082246d257f65d868ac8f549b"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.723292 25064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:06.726106 25079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.727675 25064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:06.727813 25064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "3162ac0d7b6945b69080a39f08ad88be"
format_stamp: "Formatted at 2026-06-19 06:53:06 on dist-test-slave-g8zg"
server_key: "e9b782b765b2a4f82e45e94c32bbf5b7"
server_key_iv: "5e059fd082246d257f65d868ac8f549b"
server_key_version: "encryptionkey@0"
I20260619 06:53:06.727989 25064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:06.744371 25064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:06.747720 25064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:06.747881 25064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:06.749447 25064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:06.749503 25064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.749569 25064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:06.749603 25064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:06.761240 25064 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:32965
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:06.761332 25194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:32965 every 8 connection(s)
I20260619 06:53:06.762209 25064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:06.766297 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.750675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:55501 (local address 127.22.113.62:42579)
0619 06:53:06.750864 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.750871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.752046 (+ 1175us) server_negotiation.cc:407] Connection header received
0619 06:53:06.753546 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.753550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.753616 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.753719 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.755607 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.756550 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.758395 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.758620 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.758713 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.762115 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.762119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.762122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.762165 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.763996 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.764569 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.764572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.764574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.764664 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.765026 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.765029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.765031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.765264 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.765380 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.765947 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.766093 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:53:06.767313 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.751159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.3:55501)
0619 06:53:06.751759 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.751882 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.753307 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.754001 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.754015 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.754575 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.755382 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.755406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.756714 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.756721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.758241 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.758257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.758804 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.758810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.759056 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.759860 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.759886 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.761898 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.764132 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.764138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.764156 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.764449 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.764758 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.764763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.764765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.764917 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.765377 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.765385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.765674 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.765881 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.766275 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":158,"threads_started":1}
I20260619 06:53:06.768746 25195 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42579
I20260619 06:53:06.768909 25195 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:06.769603 25195 heartbeater.cc:511] Master 127.22.113.62:42579 requested a full tablet report, sending...
I20260619 06:53:06.769892 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25064
I20260619 06:53:06.770028 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:06.770376 22980 external_mini_cluster.cc:1428] Setting key c39da89d4f988ed2046fc3661891df9d
I20260619 06:53:06.770998 24711 ts_manager.cc:194] Registered new tserver with Master: 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:06.771947 24711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:55501
I20260619 06:53:06.772815 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:53:06.780324 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.771601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.22.113.3:51171)
0619 06:53:06.771975 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.771992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.772145 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.772514 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.772516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.772785 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:06.772917 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.772925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.773997 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.774000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.774682 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.774691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.775312 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.775318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.775449 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.776067 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:06.776086 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:06.776581 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:06.778449 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.778455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.778459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.778765 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.779193 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:06.779195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:06.779196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:06.779339 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:06.779862 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:06.779864 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:06.779950 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.780039 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.780168 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260619 06:53:06.780609 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.771628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:51171 (local address 127.22.113.62:42579)
0619 06:53:06.771862 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.771867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.772072 (+ 205us) server_negotiation.cc:407] Connection header received
0619 06:53:06.772258 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.772263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.772320 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.772372 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.773046 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.773838 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.774820 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.775130 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.775218 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.776735 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.776738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.776741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.776763 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.778329 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.778911 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.778916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.778918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.779046 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.779455 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.779461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.779465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.779742 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.779877 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.780148 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.780344 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:53:06 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:06.803996 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.794866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.22.113.62:42579)
0619 06:53:06.795006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.795010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.795022 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:06.795066 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.795068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.795108 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.795214 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:06.795989 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.796542 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.797400 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.797621 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.797746 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.800677 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:06.800680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:06.800683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:06.800707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:06.802116 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.802650 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.802653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.802654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.802733 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:06.803044 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:06.803046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:06.803047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:06.803232 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:06.803407 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.803736 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.803853 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:53:06.807709 24712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:06.811364 24712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:06.828612 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.822609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.0.0.1:49698)
0619 06:53:06.822873 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.822886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.822979 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.823756 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.823759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.823777 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:06.823990 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.823998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825283 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.825289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.826895 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.826904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.827973 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.827977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.828142 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.828185 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.828309 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.828460 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":114,"thread_start_us":124,"threads_started":1}
I20260619 06:53:06.828609 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.822815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.22.113.1:33561)
0619 06:53:06.823337 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.823344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.823365 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:53:06.823438 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.823442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.823610 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.823921 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:06.824104 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825169 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.827063 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.827842 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.827973 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.828195 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.828341 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.828448 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":103,"threads_started":1}
I20260619 06:53:06.828833 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.821474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:32965 (local address 127.0.0.1:37628)
0619 06:53:06.822450 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.822494 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.822628 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.823621 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.823624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.823645 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:06.823857 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.823866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825427 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.825430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.826893 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.826904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.828364 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.828368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.828488 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.828504 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.828548 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.828615 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":132,"spinlock_wait_cycles":896,"thread_start_us":116,"threads_started":1}
I20260619 06:53:06.829321 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.821430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:37043 (local address 127.0.0.1:42848)
0619 06:53:06.822423 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.822485 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.822824 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.823601 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.823608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.823632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:06.823873 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.823879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825380 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.825386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.826496 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.826505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.828220 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.828224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.828347 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.828986 (+ 639us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.829070 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.829200 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":759,"spinlock_wait_cycles":9472,"thread_start_us":121,"threads_started":1}
I20260619 06:53:06.829387 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.821605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.22.113.2:37043)
0619 06:53:06.822109 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.822123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.822586 (+ 463us) server_negotiation.cc:407] Connection header received
0619 06:53:06.823026 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.823032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.823228 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.823409 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:06.824005 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825230 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.826623 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.827338 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.827449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.827627 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.829115 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.829224 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":127,"threads_started":1}
I20260619 06:53:06.830005 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.822405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.22.113.3:32965)
0619 06:53:06.823109 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.823123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.823147 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:53:06.823233 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.823238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.823470 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.823642 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:06.824014 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.825309 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.827357 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.828236 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.829545 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.829718 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.829770 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.829877 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":144,"threads_started":1}
I20260619 06:53:06.832279 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1791b36b1f43618a44438eadb2e06e (DEFAULT_TABLE table=test-table [id=9ba77593cc2a4e2cb5a17828131432e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:06.832530 24989 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1791b36b1f43618a44438eadb2e06e (DEFAULT_TABLE table=test-table [id=9ba77593cc2a4e2cb5a17828131432e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:06.832989 25129 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1791b36b1f43618a44438eadb2e06e (DEFAULT_TABLE table=test-table [id=9ba77593cc2a4e2cb5a17828131432e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:06.833459 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1791b36b1f43618a44438eadb2e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:06.833459 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1791b36b1f43618a44438eadb2e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:06.833914 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1791b36b1f43618a44438eadb2e06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:06.841305 25218 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Bootstrap starting.
I20260619 06:53:06.842866 25220 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Bootstrap starting.
I20260619 06:53:06.843647 25219 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Bootstrap starting.
I20260619 06:53:06.844156 25218 tablet_bootstrap.cc:654] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:06.845072 25218 log.cc:824] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:06.845346 25220 tablet_bootstrap.cc:654] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:06.845703 25219 tablet_bootstrap.cc:654] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:06.846439 25220 log.cc:824] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:06.846494 25219 log.cc:824] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:06.847668 25218 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: No bootstrap required, opened a new log
I20260619 06:53:06.847934 25218 ts_tablet_manager.cc:1397] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:53:06.848886 25219 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: No bootstrap required, opened a new log
I20260619 06:53:06.848908 25220 tablet_bootstrap.cc:492] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: No bootstrap required, opened a new log
I20260619 06:53:06.849104 25219 ts_tablet_manager.cc:1397] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 06:53:06.849104 25220 ts_tablet_manager.cc:1397] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:53:06.851872 25218 raft_consensus.cc:348] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.852157 25218 raft_consensus.cc:374] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:06.852242 25218 raft_consensus.cc:729] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Initialized, Role: FOLLOWER
I20260619 06:53:06.852797 25218 consensus_queue.cc:260] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.853554 25219 raft_consensus.cc:348] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.853534 25220 raft_consensus.cc:348] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.853832 25220 raft_consensus.cc:374] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:06.853895 25219 raft_consensus.cc:374] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:06.853915 25220 raft_consensus.cc:729] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Initialized, Role: FOLLOWER
I20260619 06:53:06.854001 25219 raft_consensus.cc:729] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Initialized, Role: FOLLOWER
I20260619 06:53:06.854002 25218 ts_tablet_manager.cc:1428] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:53:06.854453 24915 heartbeater.cc:503] Master 127.22.113.62:42579 was elected leader, sending a full tablet report...
I20260619 06:53:06.854411 25219 consensus_queue.cc:260] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.854411 25220 consensus_queue.cc:260] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.855144 25055 heartbeater.cc:503] Master 127.22.113.62:42579 was elected leader, sending a full tablet report...
I20260619 06:53:06.855396 25195 heartbeater.cc:503] Master 127.22.113.62:42579 was elected leader, sending a full tablet report...
I20260619 06:53:06.855525 25220 ts_tablet_manager.cc:1428] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:53:06.856653 25219 ts_tablet_manager.cc:1428] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:53:06.965039 25226 raft_consensus.cc:482] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:06.965297 25226 raft_consensus.cc:504] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.967038 25226 leader_election.cc:288] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965), e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561)
I20260619 06:53:06.973315 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.967841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57057 (local address 127.22.113.1:33561)
0619 06:53:06.967974 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.967980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.968352 (+ 372us) server_negotiation.cc:407] Connection header received
0619 06:53:06.968599 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.968602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.968689 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.968814 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:06.969180 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.969844 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.971751 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.972417 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.972563 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.972628 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.973054 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.973195 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:53:06.973315 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.967607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:56717 (local address 127.22.113.3:32965)
0619 06:53:06.967793 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:06.967798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:06.967840 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:53:06.968077 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:06.968081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:06.968148 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:06.968251 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:06.968676 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.969494 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.971746 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.972413 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.972563 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.972629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:06.973054 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 06:53:06.973150 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:53:06.973345 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.967587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:32965 (local address 127.22.113.2:56717)
0619 06:53:06.967755 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.967775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.967892 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.968329 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.968334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.968371 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:06.968547 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.968556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.969635 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.969638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.971576 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.971588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.972571 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.972575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.972842 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.972896 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.972988 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.973157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":155,"spinlock_wait_cycles":32512}
I20260619 06:53:06.973349 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:06.967627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.22.113.2:57057)
0619 06:53:06.968238 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:06.968255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:06.968427 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:06.968908 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:06.968911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:06.968936 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:06.969038 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.969043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.969985 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.969989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.971579 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:06.971588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:06.972571 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:06.972575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:06.972839 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:06.972891 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 06:53:06.972984 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:53:06.973157 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":303,"thread_start_us":155,"threads_started":1}
I20260619 06:53:06.974355 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1791b36b1f43618a44438eadb2e06e" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be" is_pre_election: true
I20260619 06:53:06.974324 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1791b36b1f43618a44438eadb2e06e" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be" is_pre_election: true
I20260619 06:53:06.974717 24869 raft_consensus.cc:2454] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:06.974738 25149 raft_consensus.cc:2454] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:06.975339 24940 leader_election.cc:302] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3162ac0d7b6945b69080a39f08ad88be, 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:06.975680 25226 raft_consensus.cc:2781] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:06.975800 25226 raft_consensus.cc:482] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:06.975843 25226 raft_consensus.cc:3037] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:06.977377 25226 raft_consensus.cc:504] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.977813 25226 leader_election.cc:288] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Requested vote from peers 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965), e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561)
I20260619 06:53:06.978276 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1791b36b1f43618a44438eadb2e06e" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be"
I20260619 06:53:06.978416 25149 raft_consensus.cc:3037] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:06.979447 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1791b36b1f43618a44438eadb2e06e" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be"
I20260619 06:53:06.979620 24869 raft_consensus.cc:3037] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:06.979946 25149 raft_consensus.cc:2454] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:06.980387 24943 leader_election.cc:302] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3162ac0d7b6945b69080a39f08ad88be, 8d3a04d80eb14efdb8be4de61c2236b8; no voters:
I20260619 06:53:06.980659 25226 raft_consensus.cc:2781] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:06.980945 25226 raft_consensus.cc:686] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Running, Role: LEADER
I20260619 06:53:06.981122 24869 raft_consensus.cc:2454] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:06.981372 25226 consensus_queue.cc:237] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } }
I20260619 06:53:06.985087 24711 catalog_manager.cc:5495] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } health_report { overall_health: UNKNOWN } } }
W20260619 06:53:07.013797 25196 tablet.cc:2369] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:07.022526 25056 tablet.cc:2369] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:07.037358 24916 tablet.cc:2369] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:53:07 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851985, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:07.055056 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.045238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42579 (local address 127.0.0.1:43884)
0619 06:53:07.045595 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.045610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.045778 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.046157 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.046159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.046572 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:07.046702 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.046708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.047831 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.047834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.048439 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.048447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.048931 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.048934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.048999 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.049646 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:07.049661 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:07.051270 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:07.053398 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:07.053407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:07.053412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:07.053687 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:07.053970 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:07.053972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:07.053973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:07.054063 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:07.054440 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:07.054445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:07.054562 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.054679 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.054824 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":154,"threads_started":1}
I20260619 06:53:07.055058 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.045271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.22.113.62:42579)
0619 06:53:07.045468 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.045474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.045744 (+ 270us) server_negotiation.cc:407] Connection header received
0619 06:53:07.045879 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.045882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.045952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.046076 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:07.046892 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.047706 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.048577 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.048816 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.048910 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.051446 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:07.051450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:07.051452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:07.051476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:07.053274 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:07.053799 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:07.053801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:07.053803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:07.053875 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:07.054156 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:07.054159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:07.054160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:07.054338 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:07.054416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.054744 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.054881 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":90}
I20260619 06:53:07.060060 24711 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:43884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:53:07.060833 24711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:07.066330 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 76cb2ff5647f44e288a948106bf4ddcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d925f76d4a046a788338c32e4d557bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:07.066327 24989 tablet_service.cc:1467] Processing CreateTablet for tablet 76cb2ff5647f44e288a948106bf4ddcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d925f76d4a046a788338c32e4d557bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:07.066442 25129 tablet_service.cc:1467] Processing CreateTablet for tablet 76cb2ff5647f44e288a948106bf4ddcc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d925f76d4a046a788338c32e4d557bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:07.067147 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cb2ff5647f44e288a948106bf4ddcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.067147 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cb2ff5647f44e288a948106bf4ddcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.067229 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cb2ff5647f44e288a948106bf4ddcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.070310 25220 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: Bootstrap starting.
I20260619 06:53:07.070310 25218 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: Bootstrap starting.
I20260619 06:53:07.071358 25218 tablet_bootstrap.cc:654] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.071358 25220 tablet_bootstrap.cc:654] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.071403 25219 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: Bootstrap starting.
I20260619 06:53:07.072476 25219 tablet_bootstrap.cc:654] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.072933 25220 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: No bootstrap required, opened a new log
I20260619 06:53:07.072957 25218 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: No bootstrap required, opened a new log
I20260619 06:53:07.073050 25220 ts_tablet_manager.cc:1397] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 06:53:07.073104 25218 ts_tablet_manager.cc:1397] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:53:07.073568 25220 raft_consensus.cc:348] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.073638 25218 raft_consensus.cc:348] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.073709 25220 raft_consensus.cc:374] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.073782 25218 raft_consensus.cc:374] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.073796 25220 raft_consensus.cc:729] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.073818 25218 raft_consensus.cc:729] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.073956 25218 consensus_queue.cc:260] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.074009 25219 tablet_bootstrap.cc:492] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: No bootstrap required, opened a new log
I20260619 06:53:07.073971 25220 consensus_queue.cc:260] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.074100 25219 ts_tablet_manager.cc:1397] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:07.074282 25218 ts_tablet_manager.cc:1428] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:07.074306 25228 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.074441 25228 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.074594 25219 raft_consensus.cc:348] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.074700 25219 raft_consensus.cc:374] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.074702 25220 ts_tablet_manager.cc:1428] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:53:07.074733 25219 raft_consensus.cc:729] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.074854 25219 consensus_queue.cc:260] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.075116 25224 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.075333 25224 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.075556 25225 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.075722 25225 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.075963 25219 ts_tablet_manager.cc:1428] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:53:07.230501 25224 raft_consensus.cc:482] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:07.230707 25224 raft_consensus.cc:504] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.232344 25224 leader_election.cc:288] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:07.236697 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.232925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:37513 (local address 127.22.113.3:32965)
0619 06:53:07.233130 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.233134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.233278 (+ 144us) server_negotiation.cc:407] Connection header received
0619 06:53:07.233464 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.233466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.233515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.233603 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:07.233906 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.234435 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.235689 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.236092 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.236233 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.236297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.236470 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.236540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 06:53:07.236721 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.232697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:32965 (local address 127.22.113.1:37513)
0619 06:53:07.233162 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.233176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.233300 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.233617 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.233620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.233639 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:07.233792 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.233798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.234546 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.234552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.235568 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.235578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.236234 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.236238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.236340 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.236365 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.236456 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.236540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":24,"thread_start_us":194,"threads_started":1}
I20260619 06:53:07.237224 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.232751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40371 (local address 127.22.113.2:37043)
0619 06:53:07.232970 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.232975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.233348 (+ 373us) server_negotiation.cc:407] Connection header received
0619 06:53:07.233544 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.233549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.233602 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.233702 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:07.234493 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.235043 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.236177 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.236599 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.236749 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.236811 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.236993 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.237078 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:53:07.237433 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cb2ff5647f44e288a948106bf4ddcc" candidate_uuid: "e1864da7500a4c6b833b5c99561709be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be" is_pre_election: true
I20260619 06:53:07.237643 25149 raft_consensus.cc:2454] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1864da7500a4c6b833b5c99561709be in term 0.
I20260619 06:53:07.237717 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.232708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:37043 (local address 127.22.113.1:40371)
0619 06:53:07.233226 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.233264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.233411 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.233715 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.233718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.233734 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:07.234362 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.234370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.235172 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.235177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.236040 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.236048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.236737 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.236741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.236854 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.236878 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.237418 (+ 540us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.237509 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":38,"thread_start_us":161,"threads_started":1}
I20260619 06:53:07.238098 24803 leader_election.cc:302] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3162ac0d7b6945b69080a39f08ad88be, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:07.238500 25224 raft_consensus.cc:2781] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:07.238575 25224 raft_consensus.cc:482] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:07.238613 25224 raft_consensus.cc:3037] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.238514 25009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cb2ff5647f44e288a948106bf4ddcc" candidate_uuid: "e1864da7500a4c6b833b5c99561709be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" is_pre_election: true
I20260619 06:53:07.238802 25009 raft_consensus.cc:2454] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1864da7500a4c6b833b5c99561709be in term 0.
I20260619 06:53:07.240043 25224 raft_consensus.cc:504] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.240494 25224 leader_election.cc:288] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [CANDIDATE]: Term 1 election: Requested vote from peers 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:07.240891 25009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cb2ff5647f44e288a948106bf4ddcc" candidate_uuid: "e1864da7500a4c6b833b5c99561709be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3a04d80eb14efdb8be4de61c2236b8"
I20260619 06:53:07.241004 25009 raft_consensus.cc:3037] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.240938 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cb2ff5647f44e288a948106bf4ddcc" candidate_uuid: "e1864da7500a4c6b833b5c99561709be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be"
I20260619 06:53:07.241097 25149 raft_consensus.cc:3037] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.242077 25009 raft_consensus.cc:2454] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1864da7500a4c6b833b5c99561709be in term 1.
I20260619 06:53:07.242254 25149 raft_consensus.cc:2454] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1864da7500a4c6b833b5c99561709be in term 1.
I20260619 06:53:07.242383 24801 leader_election.cc:302] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:07.242599 25224 raft_consensus.cc:2781] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:07.242828 25224 raft_consensus.cc:686] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [term 1 LEADER]: Becoming Leader. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Running, Role: LEADER
I20260619 06:53:07.243311 25224 consensus_queue.cc:237] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.246053 25241 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: TxnStatusTablet state changed. Reason: New leader e1864da7500a4c6b833b5c99561709be. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.246245 25241 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:07.246618 25243 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:53:07.246887 25243 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:53:07.247668 24710 catalog_manager.cc:5495] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be reported cstate change: term changed from 0 to 1, leader changed from <none> to e1864da7500a4c6b833b5c99561709be (127.22.113.1). New cstate: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:07.269498 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.264954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.0.0.1:49700)
0619 06:53:07.265239 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.265255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.265372 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.265737 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.265739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.265746 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:07.265862 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.265867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.266886 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.266888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.267616 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.267625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.268269 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.268272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.268347 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.268398 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.269209 (+ 811us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.269225 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.269287 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.269363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 06:53:07.269544 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.265103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.22.113.1:33561)
0619 06:53:07.265257 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.265262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.265305 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:53:07.265535 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.265539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.265606 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.265720 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:07.265971 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.266776 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.267796 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.268152 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.268259 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.268546 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.269102 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.269244 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.269317 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.269376 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:53:07.275748 25149 raft_consensus.cc:1261] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Refusing update from remote peer e1864da7500a4c6b833b5c99561709be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.275745 25009 raft_consensus.cc:1261] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Refusing update from remote peer e1864da7500a4c6b833b5c99561709be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.276500 25224 consensus_queue.cc:1035] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:07.276741 25241 consensus_queue.cc:1035] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:07.281040 25228 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: TxnStatusTablet state changed. Reason: New leader e1864da7500a4c6b833b5c99561709be. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.281178 25228 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.281764 25225 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: TxnStatusTablet state changed. Reason: New leader e1864da7500a4c6b833b5c99561709be. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.281893 25225 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.285072 25224 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.285197 25224 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:07.285332 25246 mvcc.cc:205] Tried to move back new op lower bound from 7298465739873460224 to 7298465739751747584. Current Snapshot: MvccSnapshot[applied={T|T < 7298465739873460224}]
I20260619 06:53:07.286377 25225 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.286484 25225 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 3162ac0d7b6945b69080a39f08ad88be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.288002 25228 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.288146 25228 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P 8d3a04d80eb14efdb8be4de61c2236b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:07.288733 25224 tablet_replica.cc:439] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1864da7500a4c6b833b5c99561709be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } } }
I20260619 06:53:07.288872 25224 tablet_replica.cc:442] T 76cb2ff5647f44e288a948106bf4ddcc P e1864da7500a4c6b833b5c99561709be: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:07.293138 25248 mvcc.cc:205] Tried to move back new op lower bound from 7298465739873460224 to 7298465739751747584. Current Snapshot: MvccSnapshot[applied={T|T < 7298465739873460224 or (T in {7298465739873460224})}]
I20260619 06:53:07.293449 25247 mvcc.cc:205] Tried to move back new op lower bound from 7298465739873460224 to 7298465739751747584. Current Snapshot: MvccSnapshot[applied={T|T < 7298465739873460224 or (T in {7298465739873460224})}]
I20260619 06:53:07.305444 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.300541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.22.113.2:37043)
0619 06:53:07.300688 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.300694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.300826 (+ 132us) server_negotiation.cc:407] Connection header received
0619 06:53:07.301024 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.301027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.301104 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.301242 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:07.301546 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.302379 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.303723 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.303949 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.304038 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.304399 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.304979 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.305076 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.305189 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.305244 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:53:07.318560 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.314693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:34301 (local address 127.22.113.1:33561)
0619 06:53:07.314867 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.314872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.315009 (+ 137us) server_negotiation.cc:407] Connection header received
0619 06:53:07.315188 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.315192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.315296 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.315392 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:07.315696 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.316304 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.317304 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.317584 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.317702 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.318035 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.318125 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.318208 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.318329 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.318391 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:53:07.318576 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.314575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.22.113.2:34301)
0619 06:53:07.314904 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.314921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.315073 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.315402 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.315406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.315413 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:07.315596 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.315605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.316418 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.316424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.317175 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.317184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.317703 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.317707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.317823 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.317870 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.318211 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.318220 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.318304 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.318393 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
I20260619 06:53:07.326985 25149 raft_consensus.cc:1261] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.327155 24869 raft_consensus.cc:1261] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.327612 25228 consensus_queue.cc:1035] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:07.327805 25226 consensus_queue.cc:1035] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:07.353942 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.349300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:37043 (local address 127.22.113.1:44363)
0619 06:53:07.349681 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:07.349698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:07.349817 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:07.350127 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:07.350130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:07.350138 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:07.350253 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.350259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.351309 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.351313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.352256 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:07.352266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.353022 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.353025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:07.353098 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.353193 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.353579 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.353587 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:53:07.353657 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:53:07.353766 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":138,"threads_started":1}
I20260619 06:53:07.354408 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:07.349398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44363 (local address 127.22.113.2:37043)
0619 06:53:07.349558 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:07.349565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:07.349783 (+ 218us) server_negotiation.cc:407] Connection header received
0619 06:53:07.349914 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:07.349942 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:07.350003 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:07.350109 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:07.350366 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.351137 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.352426 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:07.352843 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:07.352974 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:07.353348 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:07.353446 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:07.353533 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:07.353869 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:53:07.353945 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:53:07.383736 24710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:07.384004 24710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:07.387652 24710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Sending DeleteTablet for 3 replicas of tablet 0c1791b36b1f43618a44438eadb2e06e
I20260619 06:53:07.388919 24989 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1791b36b1f43618a44438eadb2e06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:07 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42848
I20260619 06:53:07.388918 25129 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1791b36b1f43618a44438eadb2e06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:07 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37628
I20260619 06:53:07.389576 25267 tablet_replica.cc:330] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: stopping tablet replica
I20260619 06:53:07.389926 24849 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1791b36b1f43618a44438eadb2e06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:07 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:49698
I20260619 06:53:07.389987 25267 raft_consensus.cc:2229] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:07.390267 25267 raft_consensus.cc:2258] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:07.391316 24710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:07.391856 24710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:07.392168 25267 ts_tablet_manager.cc:1905] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:07.393187 25268 tablet_replica.cc:330] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: stopping tablet replica
I20260619 06:53:07.393615 25268 raft_consensus.cc:2229] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:07.394093 25268 raft_consensus.cc:2258] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:07.396848 25268 ts_tablet_manager.cc:1905] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:07.398156 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 3172a6ea3e414eb8b29772566108de6a (DEFAULT_TABLE table=test-table [id=657a9d20403a41eaa4536614dcf416a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:07.398481 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3172a6ea3e414eb8b29772566108de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.398556 25129 tablet_service.cc:1467] Processing CreateTablet for tablet 3172a6ea3e414eb8b29772566108de6a (DEFAULT_TABLE table=test-table [id=657a9d20403a41eaa4536614dcf416a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:07.398910 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3172a6ea3e414eb8b29772566108de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.399583 24989 tablet_service.cc:1467] Processing CreateTablet for tablet 3172a6ea3e414eb8b29772566108de6a (DEFAULT_TABLE table=test-table [id=657a9d20403a41eaa4536614dcf416a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:07.401747 25219 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Bootstrap starting.
I20260619 06:53:07.402840 25219 tablet_bootstrap.cc:654] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.403431 25218 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Bootstrap starting.
I20260619 06:53:07.404457 25218 tablet_bootstrap.cc:654] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.406543 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3172a6ea3e414eb8b29772566108de6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:07.406697 25267 ts_tablet_manager.cc:1918] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:07.406800 25267 log.cc:1196] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/0c1791b36b1f43618a44438eadb2e06e
I20260619 06:53:07.407196 25267 ts_tablet_manager.cc:1939] T 0c1791b36b1f43618a44438eadb2e06e P 3162ac0d7b6945b69080a39f08ad88be: Deleting consensus metadata
I20260619 06:53:07.407516 25269 tablet_replica.cc:330] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: stopping tablet replica
I20260619 06:53:07.407934 25269 raft_consensus.cc:2229] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:07.408154 25269 raft_consensus.cc:2258] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:07.409063 25220 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Bootstrap starting.
I20260619 06:53:07.409973 25220 tablet_bootstrap.cc:654] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:07.409929 24699 catalog_manager.cc:4841] TS 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965): tablet 0c1791b36b1f43618a44438eadb2e06e (table test-table [id=9ba77593cc2a4e2cb5a17828131432e6]) successfully deleted
I20260619 06:53:07.411423 25269 ts_tablet_manager.cc:1905] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:07.414795 25269 ts_tablet_manager.cc:1918] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:07.414904 25269 log.cc:1196] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/0c1791b36b1f43618a44438eadb2e06e
I20260619 06:53:07.415900 25220 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: No bootstrap required, opened a new log
I20260619 06:53:07.415992 25220 ts_tablet_manager.cc:1397] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260619 06:53:07.416178 25219 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: No bootstrap required, opened a new log
I20260619 06:53:07.416281 25219 ts_tablet_manager.cc:1397] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20260619 06:53:07.416493 25220 raft_consensus.cc:348] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.416600 25220 raft_consensus.cc:374] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.416630 25220 raft_consensus.cc:729] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.416746 25220 consensus_queue.cc:260] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.416813 25219 raft_consensus.cc:348] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.416939 25219 raft_consensus.cc:374] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.416970 25219 raft_consensus.cc:729] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.416990 25220 ts_tablet_manager.cc:1428] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:07.417112 25219 consensus_queue.cc:260] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.417388 25219 ts_tablet_manager.cc:1428] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:07.419889 25268 ts_tablet_manager.cc:1918] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:07.420389 25268 log.cc:1196] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/0c1791b36b1f43618a44438eadb2e06e
I20260619 06:53:07.420806 25268 ts_tablet_manager.cc:1939] T 0c1791b36b1f43618a44438eadb2e06e P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting consensus metadata
I20260619 06:53:07.420982 25269 ts_tablet_manager.cc:1939] T 0c1791b36b1f43618a44438eadb2e06e P e1864da7500a4c6b833b5c99561709be: Deleting consensus metadata
I20260619 06:53:07.421324 25218 tablet_bootstrap.cc:492] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: No bootstrap required, opened a new log
I20260619 06:53:07.421427 25218 ts_tablet_manager.cc:1397] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20260619 06:53:07.421941 25218 raft_consensus.cc:348] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.422083 25218 raft_consensus.cc:374] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:07.422123 25218 raft_consensus.cc:729] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Initialized, Role: FOLLOWER
I20260619 06:53:07.422256 25218 consensus_queue.cc:260] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.422526 25218 ts_tablet_manager.cc:1428] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:07.422899 24696 catalog_manager.cc:4841] TS e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561): tablet 0c1791b36b1f43618a44438eadb2e06e (table test-table [id=9ba77593cc2a4e2cb5a17828131432e6]) successfully deleted
I20260619 06:53:07.424609 24697 catalog_manager.cc:4841] TS 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043): tablet 0c1791b36b1f43618a44438eadb2e06e (table test-table [id=9ba77593cc2a4e2cb5a17828131432e6]) successfully deleted
I20260619 06:53:07.673097 25261 raft_consensus.cc:482] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:07.673260 25261 raft_consensus.cc:504] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.673779 25261 leader_election.cc:288] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:07.674270 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3172a6ea3e414eb8b29772566108de6a" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be" is_pre_election: true
I20260619 06:53:07.674461 24869 raft_consensus.cc:2454] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:07.674556 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3172a6ea3e414eb8b29772566108de6a" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be" is_pre_election: true
I20260619 06:53:07.674816 25149 raft_consensus.cc:2454] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:07.674866 24940 leader_election.cc:302] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:07.675092 25261 raft_consensus.cc:2781] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:07.675168 25261 raft_consensus.cc:482] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:07.675197 25261 raft_consensus.cc:3037] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.676501 25261 raft_consensus.cc:504] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.676949 25261 leader_election.cc:288] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Requested vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:07.677412 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3172a6ea3e414eb8b29772566108de6a" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be"
I20260619 06:53:07.677573 24869 raft_consensus.cc:3037] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.677520 25149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3172a6ea3e414eb8b29772566108de6a" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be"
I20260619 06:53:07.677690 25149 raft_consensus.cc:3037] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:07.678699 24869 raft_consensus.cc:2454] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:07.678913 25149 raft_consensus.cc:2454] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:07.679040 24940 leader_election.cc:302] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:07.679217 25261 raft_consensus.cc:2781] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:07.679335 25261 raft_consensus.cc:686] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Running, Role: LEADER
I20260619 06:53:07.679512 25261 consensus_queue.cc:237] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:07.681758 24712 catalog_manager.cc:5495] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:07.714452 24869 raft_consensus.cc:1261] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.714493 25149 raft_consensus.cc:1261] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:07.715014 25226 consensus_queue.cc:1035] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:07.715219 25261 consensus_queue.cc:1035] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:07.729854 25247 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:53:09.292729 24712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:09.292917 24712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:09.296096 24712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Sending DeleteTablet for 3 replicas of tablet 3172a6ea3e414eb8b29772566108de6a
I20260619 06:53:09.296873 24849 tablet_service.cc:1514] Processing DeleteTablet for tablet 3172a6ea3e414eb8b29772566108de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:49698
I20260619 06:53:09.296973 25129 tablet_service.cc:1514] Processing DeleteTablet for tablet 3172a6ea3e414eb8b29772566108de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37628
I20260619 06:53:09.297293 24989 tablet_service.cc:1514] Processing DeleteTablet for tablet 3172a6ea3e414eb8b29772566108de6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42848
I20260619 06:53:09.297323 25300 tablet_replica.cc:330] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: stopping tablet replica
I20260619 06:53:09.297457 25300 raft_consensus.cc:2229] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:09.297564 25300 raft_consensus.cc:2258] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.298214 25300 ts_tablet_manager.cc:1905] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.298656 25301 tablet_replica.cc:330] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: stopping tablet replica
I20260619 06:53:09.298807 25301 raft_consensus.cc:2229] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:09.298928 25301 raft_consensus.cc:2258] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.299612 25301 ts_tablet_manager.cc:1905] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.301056 25300 ts_tablet_manager.cc:1918] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:53:09.301187 25300 log.cc:1196] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/3172a6ea3e414eb8b29772566108de6a
I20260619 06:53:09.301321 25302 tablet_replica.cc:330] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: stopping tablet replica
I20260619 06:53:09.301524 25300 ts_tablet_manager.cc:1939] T 3172a6ea3e414eb8b29772566108de6a P e1864da7500a4c6b833b5c99561709be: Deleting consensus metadata
I20260619 06:53:09.301558 25302 raft_consensus.cc:2229] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:09.301872 25302 raft_consensus.cc:2258] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.302424 24696 catalog_manager.cc:4841] TS e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561): tablet 3172a6ea3e414eb8b29772566108de6a (table test-table [id=657a9d20403a41eaa4536614dcf416a1]) successfully deleted
I20260619 06:53:09.302863 25302 ts_tablet_manager.cc:1905] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.304015 25301 ts_tablet_manager.cc:1918] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:53:09.304112 25301 log.cc:1196] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/3172a6ea3e414eb8b29772566108de6a
I20260619 06:53:09.304457 25301 ts_tablet_manager.cc:1939] T 3172a6ea3e414eb8b29772566108de6a P 3162ac0d7b6945b69080a39f08ad88be: Deleting consensus metadata
I20260619 06:53:09.305389 24699 catalog_manager.cc:4841] TS 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965): tablet 3172a6ea3e414eb8b29772566108de6a (table test-table [id=657a9d20403a41eaa4536614dcf416a1]) successfully deleted
I20260619 06:53:09.305984 25302 ts_tablet_manager.cc:1918] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:53:09.306092 25302 log.cc:1196] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/3172a6ea3e414eb8b29772566108de6a
I20260619 06:53:09.306490 25302 ts_tablet_manager.cc:1939] T 3172a6ea3e414eb8b29772566108de6a P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting consensus metadata
I20260619 06:53:09.307507 24697 catalog_manager.cc:4841] TS 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043): tablet 3172a6ea3e414eb8b29772566108de6a (table test-table [id=657a9d20403a41eaa4536614dcf416a1]) successfully deleted
Jun 19 06:53:09 dist-test-slave-g8zg krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:53:09 dist-test-slave-g8zg krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:09.332823 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.318914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.22.113.62:42579)
0619 06:53:09.319205 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.319212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.319392 (+ 180us) server_negotiation.cc:407] Connection header received
0619 06:53:09.319487 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.319489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.319543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.319655 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:09.320503 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.321066 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.322070 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.322345 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.322439 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.329413 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:09.329416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:09.329418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:09.329451 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:09.330912 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:09.331474 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:09.331477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:09.331478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:09.331578 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:09.331854 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:09.331856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:09.331858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:09.332043 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:09.332147 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.332528 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.332641 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":125,"threads_started":1}
I20260619 06:53:09.336213 24712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:09.336822 24712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:09.342474 24989 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 (DEFAULT_TABLE table=test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.342482 24849 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 (DEFAULT_TABLE table=test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.342837 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58c6bd62a24e34963d5c005ba206a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.342868 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58c6bd62a24e34963d5c005ba206a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.342806 25129 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 (DEFAULT_TABLE table=test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.343106 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58c6bd62a24e34963d5c005ba206a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.345557 25311 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Bootstrap starting.
I20260619 06:53:09.346062 25313 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Bootstrap starting.
I20260619 06:53:09.346349 25312 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Bootstrap starting.
I20260619 06:53:09.346740 25311 tablet_bootstrap.cc:654] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.347080 25313 tablet_bootstrap.cc:654] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.347739 25312 tablet_bootstrap.cc:654] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.348284 25311 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: No bootstrap required, opened a new log
I20260619 06:53:09.348412 25311 ts_tablet_manager.cc:1397] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:53:09.348721 25313 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: No bootstrap required, opened a new log
I20260619 06:53:09.348893 25313 ts_tablet_manager.cc:1397] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:09.348974 25312 tablet_bootstrap.cc:492] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: No bootstrap required, opened a new log
I20260619 06:53:09.348958 25311 raft_consensus.cc:348] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.349063 25312 ts_tablet_manager.cc:1397] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:09.349110 25311 raft_consensus.cc:374] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.349153 25311 raft_consensus.cc:729] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.349334 25311 consensus_queue.cc:260] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.349514 25312 raft_consensus.cc:348] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.349625 25312 raft_consensus.cc:374] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.349659 25312 raft_consensus.cc:729] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.349745 25311 ts_tablet_manager.cc:1428] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 06:53:09.349709 25313 raft_consensus.cc:348] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.349779 25312 consensus_queue.cc:260] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.349843 25313 raft_consensus.cc:374] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.349879 25313 raft_consensus.cc:729] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.350015 25313 consensus_queue.cc:260] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.350210 25312 ts_tablet_manager.cc:1428] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:09.350338 25313 ts_tablet_manager.cc:1428] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:09.694674 25317 raft_consensus.cc:482] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:09.694856 25317 raft_consensus.cc:504] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.696220 25317 leader_election.cc:288] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043)
I20260619 06:53:09.700759 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.696720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:38407 (local address 127.22.113.2:37043)
0619 06:53:09.697130 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.697142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.697163 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:53:09.697395 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.697398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.697448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.697554 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:09.697856 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.698456 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.699607 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.700056 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.700215 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.700291 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.700494 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.700579 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260619 06:53:09.700776 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.696620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:37043 (local address 127.22.113.3:38407)
0619 06:53:09.697063 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:09.697081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:09.697226 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:09.697565 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:09.697568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:09.697584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:09.697742 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.697749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.698594 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.698597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.699478 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.699486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.700215 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.700219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.700359 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.700382 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:53:09.700477 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:53:09.700571 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":2,"thread_start_us":151,"threads_started":1}
I20260619 06:53:09.700776 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.696620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:44535 (local address 127.22.113.1:33561)
0619 06:53:09.697118 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.697134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.697155 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:53:09.697228 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.697231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.697286 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.697402 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:09.697724 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.698308 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.699585 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.700051 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.700210 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.700298 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.700534 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.700610 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":163,"threads_started":1}
I20260619 06:53:09.700832 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.696457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.22.113.3:44535)
0619 06:53:09.696925 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:09.696948 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:09.697088 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:09.697418 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:09.697422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:09.697444 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:09.697595 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.697602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.698441 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.698448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.699448 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.699458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.700213 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.700218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.700387 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.700412 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:53:09.700517 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:53:09.700620 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":31,"thread_start_us":157,"threads_started":1}
I20260619 06:53:09.701280 25009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58c6bd62a24e34963d5c005ba206a6" candidate_uuid: "3162ac0d7b6945b69080a39f08ad88be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" is_pre_election: true
I20260619 06:53:09.701474 25009 raft_consensus.cc:2454] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3162ac0d7b6945b69080a39f08ad88be in term 0.
I20260619 06:53:09.701592 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58c6bd62a24e34963d5c005ba206a6" candidate_uuid: "3162ac0d7b6945b69080a39f08ad88be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be" is_pre_election: true
I20260619 06:53:09.701804 24869 raft_consensus.cc:2454] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3162ac0d7b6945b69080a39f08ad88be in term 0.
I20260619 06:53:09.702013 25081 leader_election.cc:302] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3162ac0d7b6945b69080a39f08ad88be, 8d3a04d80eb14efdb8be4de61c2236b8; no voters:
I20260619 06:53:09.702447 25317 raft_consensus.cc:2781] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:09.702533 25317 raft_consensus.cc:482] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:09.702595 25317 raft_consensus.cc:3037] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.703737 25317 raft_consensus.cc:504] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.704154 25317 leader_election.cc:288] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [CANDIDATE]: Term 1 election: Requested vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043)
I20260619 06:53:09.704589 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58c6bd62a24e34963d5c005ba206a6" candidate_uuid: "3162ac0d7b6945b69080a39f08ad88be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be"
I20260619 06:53:09.704594 25009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58c6bd62a24e34963d5c005ba206a6" candidate_uuid: "3162ac0d7b6945b69080a39f08ad88be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3a04d80eb14efdb8be4de61c2236b8"
I20260619 06:53:09.704723 24869 raft_consensus.cc:3037] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.704727 25009 raft_consensus.cc:3037] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.705958 25009 raft_consensus.cc:2454] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3162ac0d7b6945b69080a39f08ad88be in term 1.
I20260619 06:53:09.706094 24869 raft_consensus.cc:2454] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3162ac0d7b6945b69080a39f08ad88be in term 1.
I20260619 06:53:09.706327 25081 leader_election.cc:302] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3162ac0d7b6945b69080a39f08ad88be, 8d3a04d80eb14efdb8be4de61c2236b8; no voters:
I20260619 06:53:09.706561 25317 raft_consensus.cc:2781] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:09.706840 25317 raft_consensus.cc:686] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 1 LEADER]: Becoming Leader. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Running, Role: LEADER
I20260619 06:53:09.707221 25317 consensus_queue.cc:237] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.710264 24711 catalog_manager.cc:5495] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be reported cstate change: term changed from 0 to 1, leader changed from <none> to 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3). New cstate: current_term: 1 leader_uuid: "3162ac0d7b6945b69080a39f08ad88be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } health_report { overall_health: HEALTHY } } }
I20260619 06:53:09.728492 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.724180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.22.113.3:32965)
0619 06:53:09.724509 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.724533 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.724544 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:53:09.724624 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.724626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.724685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.724830 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:09.725113 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.725741 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.726638 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.726888 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.727002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.727363 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.727976 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.728115 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.728205 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.728269 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":123,"threads_started":1}
I20260619 06:53:09.740484 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.737053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:34645 (local address 127.22.113.1:33561)
0619 06:53:09.737195 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.737199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.737402 (+ 203us) server_negotiation.cc:407] Connection header received
0619 06:53:09.737528 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.737530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.737578 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.737654 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:09.737918 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.738428 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.739426 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.739648 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.739739 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.740066 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.740140 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.740199 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.740323 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.740372 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:53:09.740515 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.736930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.22.113.3:34645)
0619 06:53:09.737303 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:09.737318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:09.737428 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:09.737682 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:09.737684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:09.737690 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:09.737824 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.737828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.738530 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.738532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.739306 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.739316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.739790 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.739794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.739882 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.739933 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.740229 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.740235 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:53:09.740290 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 06:53:09.740361 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":141,"threads_started":1}
I20260619 06:53:09.747294 25009 raft_consensus.cc:1261] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Refusing update from remote peer 3162ac0d7b6945b69080a39f08ad88be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:09.747397 24869 raft_consensus.cc:1261] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Refusing update from remote peer 3162ac0d7b6945b69080a39f08ad88be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:09.747925 25322 consensus_queue.cc:1035] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:09.748122 25317 consensus_queue.cc:1035] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:09.774319 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.770094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:54667 (local address 127.22.113.3:32965)
0619 06:53:09.770224 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.770229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.770452 (+ 223us) server_negotiation.cc:407] Connection header received
0619 06:53:09.770676 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.770679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.770741 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.770849 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:09.771067 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.771898 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.773142 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.773485 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.773590 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.773887 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.773966 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.774036 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.774130 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.774180 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:53:09.774355 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.770039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:32965 (local address 127.22.113.1:54667)
0619 06:53:09.770358 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:09.770375 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:09.770532 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:09.770845 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:09.770847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:09.770853 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:09.770966 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.770971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.772048 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.772052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.773004 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:09.773013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.773593 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.773596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:09.773695 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.773756 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.774057 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.774062 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:53:09.774117 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 06:53:09.774207 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":124,"threads_started":1}
I20260619 06:53:09.798223 24711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:09.798468 24711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:09.801050 24711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Sending DeleteTablet for 3 replicas of tablet 4e58c6bd62a24e34963d5c005ba206a6
I20260619 06:53:09.801739 24989 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42848
I20260619 06:53:09.801739 24849 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:49698
I20260619 06:53:09.801959 25302 tablet_replica.cc:330] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: stopping tablet replica
I20260619 06:53:09.802083 25302 raft_consensus.cc:2229] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:09.802188 25302 raft_consensus.cc:2258] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.802556 25129 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e58c6bd62a24e34963d5c005ba206a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37628
I20260619 06:53:09.802796 25301 tablet_replica.cc:330] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: stopping tablet replica
I20260619 06:53:09.802906 25301 raft_consensus.cc:2229] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:09.803012 25302 ts_tablet_manager.cc:1905] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.802933 24711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:53:09.803212 25301 raft_consensus.cc:2258] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.803349 25300 tablet_replica.cc:330] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: stopping tablet replica
I20260619 06:53:09.803471 25300 raft_consensus.cc:2229] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 06:53:09.803481 24711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:09.803586 25300 raft_consensus.cc:2258] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:09.803956 25301 ts_tablet_manager.cc:1905] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.806416 25301 ts_tablet_manager.cc:1918] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:09.806504 25301 log.cc:1196] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/4e58c6bd62a24e34963d5c005ba206a6
I20260619 06:53:09.806690 25302 ts_tablet_manager.cc:1918] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:09.806756 25302 log.cc:1196] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/4e58c6bd62a24e34963d5c005ba206a6
I20260619 06:53:09.806855 25301 ts_tablet_manager.cc:1939] T 4e58c6bd62a24e34963d5c005ba206a6 P 3162ac0d7b6945b69080a39f08ad88be: Deleting consensus metadata
I20260619 06:53:09.806888 25300 ts_tablet_manager.cc:1905] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:09.807018 25302 ts_tablet_manager.cc:1939] T 4e58c6bd62a24e34963d5c005ba206a6 P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting consensus metadata
I20260619 06:53:09.807618 24699 catalog_manager.cc:4841] TS 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965): tablet 4e58c6bd62a24e34963d5c005ba206a6 (table test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]) successfully deleted
I20260619 06:53:09.807677 24697 catalog_manager.cc:4841] TS 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043): tablet 4e58c6bd62a24e34963d5c005ba206a6 (table test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]) successfully deleted
I20260619 06:53:09.809952 25300 ts_tablet_manager.cc:1918] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:09.810050 25300 log.cc:1196] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/4e58c6bd62a24e34963d5c005ba206a6
I20260619 06:53:09.810341 25300 ts_tablet_manager.cc:1939] T 4e58c6bd62a24e34963d5c005ba206a6 P e1864da7500a4c6b833b5c99561709be: Deleting consensus metadata
I20260619 06:53:09.811129 24849 tablet_service.cc:1467] Processing CreateTablet for tablet f94705fc8500438994d0785cf111baaf (DEFAULT_TABLE table=test-table [id=ef12579b583b434384a856151bcfec40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.811151 24696 catalog_manager.cc:4841] TS e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561): tablet 4e58c6bd62a24e34963d5c005ba206a6 (table test-table [id=6464c7874fb242b4b06d93d7c4a6dab2]) successfully deleted
I20260619 06:53:09.811365 25129 tablet_service.cc:1467] Processing CreateTablet for tablet f94705fc8500438994d0785cf111baaf (DEFAULT_TABLE table=test-table [id=ef12579b583b434384a856151bcfec40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.811461 24849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94705fc8500438994d0785cf111baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.811579 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94705fc8500438994d0785cf111baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.812106 24989 tablet_service.cc:1467] Processing CreateTablet for tablet f94705fc8500438994d0785cf111baaf (DEFAULT_TABLE table=test-table [id=ef12579b583b434384a856151bcfec40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:09.812347 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94705fc8500438994d0785cf111baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:09.813735 25312 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Bootstrap starting.
I20260619 06:53:09.814628 25313 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Bootstrap starting.
I20260619 06:53:09.814850 25312 tablet_bootstrap.cc:654] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.815384 25313 tablet_bootstrap.cc:654] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.816350 25312 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: No bootstrap required, opened a new log
I20260619 06:53:09.816447 25313 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: No bootstrap required, opened a new log
I20260619 06:53:09.816470 25312 ts_tablet_manager.cc:1397] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:53:09.816545 25313 ts_tablet_manager.cc:1397] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:53:09.816982 25313 raft_consensus.cc:348] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.816974 25312 raft_consensus.cc:348] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.817077 25313 raft_consensus.cc:374] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.817095 25312 raft_consensus.cc:374] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.817127 25312 raft_consensus.cc:729] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3162ac0d7b6945b69080a39f08ad88be, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.817099 25313 raft_consensus.cc:729] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.817258 25313 consensus_queue.cc:260] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.817538 25313 ts_tablet_manager.cc:1428] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:09.818193 25312 consensus_queue.cc:260] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.818601 25312 ts_tablet_manager.cc:1428] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:53:09.821691 25311 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: Bootstrap starting.
I20260619 06:53:09.822590 25311 tablet_bootstrap.cc:654] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:09.823807 25311 tablet_bootstrap.cc:492] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: No bootstrap required, opened a new log
I20260619 06:53:09.823917 25311 ts_tablet_manager.cc:1397] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:53:09.824268 25311 raft_consensus.cc:348] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.824398 25311 raft_consensus.cc:374] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:09.824442 25311 raft_consensus.cc:729] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1864da7500a4c6b833b5c99561709be, State: Initialized, Role: FOLLOWER
I20260619 06:53:09.824575 25311 consensus_queue.cc:260] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.824831 25311 ts_tablet_manager.cc:1428] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:09.827600 25295 raft_consensus.cc:482] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:09.827744 25295 raft_consensus.cc:504] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.828219 25295 leader_election.cc:288] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:09.828586 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94705fc8500438994d0785cf111baaf" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be" is_pre_election: true
I20260619 06:53:09.828644 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94705fc8500438994d0785cf111baaf" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be" is_pre_election: true
I20260619 06:53:09.828732 24869 raft_consensus.cc:2454] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:09.828837 25148 raft_consensus.cc:2454] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 0.
I20260619 06:53:09.829037 24940 leader_election.cc:302] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:09.829232 25295 raft_consensus.cc:2781] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:09.829288 25295 raft_consensus.cc:482] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:09.829324 25295 raft_consensus.cc:3037] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.830677 25295 raft_consensus.cc:504] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.831040 25295 leader_election.cc:288] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Requested vote from peers e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561), 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965)
I20260619 06:53:09.831646 24869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94705fc8500438994d0785cf111baaf" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1864da7500a4c6b833b5c99561709be"
I20260619 06:53:09.831805 24869 raft_consensus.cc:3037] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.833079 24869 raft_consensus.cc:2454] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:09.833380 25148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94705fc8500438994d0785cf111baaf" candidate_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3162ac0d7b6945b69080a39f08ad88be"
I20260619 06:53:09.833451 24940 leader_election.cc:302] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3a04d80eb14efdb8be4de61c2236b8, e1864da7500a4c6b833b5c99561709be; no voters:
I20260619 06:53:09.833510 25148 raft_consensus.cc:3037] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:09.833616 25295 raft_consensus.cc:2781] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:09.833797 25295 raft_consensus.cc:686] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3a04d80eb14efdb8be4de61c2236b8, State: Running, Role: LEADER
I20260619 06:53:09.833949 25295 consensus_queue.cc:237] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } }
I20260619 06:53:09.834753 25148 raft_consensus.cc:2454] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3a04d80eb14efdb8be4de61c2236b8 in term 1.
I20260619 06:53:09.835762 24710 catalog_manager.cc:5495] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d3a04d80eb14efdb8be4de61c2236b8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 37043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:09.852360 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:09.848791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.22.113.2:37043)
0619 06:53:09.849135 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:09.849139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:09.849149 (+ 10us) server_negotiation.cc:407] Connection header received
0619 06:53:09.849188 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:09.849190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:09.849234 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:09.849324 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:09.849617 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.850194 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.851145 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:09.851449 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:09.851572 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:09.851840 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:09.851919 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:09.852002 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:09.852139 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:53:09.852191 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260619 06:53:09.859927 24869 raft_consensus.cc:1261] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:09.859927 25148 raft_consensus.cc:1261] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Refusing update from remote peer 8d3a04d80eb14efdb8be4de61c2236b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:09.860343 25295 consensus_queue.cc:1035] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1864da7500a4c6b833b5c99561709be" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:09.860549 25337 consensus_queue.cc:1035] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3162ac0d7b6945b69080a39f08ad88be" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:09.878046 25282 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:53:11.439877 24710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:11.440063 24710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:11.442680 24710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a9e6e7db8e4af29e56974e6ff6175b: Sending DeleteTablet for 3 replicas of tablet f94705fc8500438994d0785cf111baaf
I20260619 06:53:11.443303 24849 tablet_service.cc:1514] Processing DeleteTablet for tablet f94705fc8500438994d0785cf111baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:11 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:49698
I20260619 06:53:11.443473 24989 tablet_service.cc:1514] Processing DeleteTablet for tablet f94705fc8500438994d0785cf111baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:11 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42848
I20260619 06:53:11.443697 25365 tablet_replica.cc:330] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be: stopping tablet replica
I20260619 06:53:11.443890 25365 raft_consensus.cc:2229] T f94705fc8500438994d0785cf111baaf P e1864da7500a4c6b833b5c99561709be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:11.443883 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24784
I20260619 06:53:11.443869 25129 tablet_service.cc:1514] Processing DeleteTablet for tablet f94705fc8500438994d0785cf111baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:11 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37628
I20260619 06:53:11.445080 25366 tablet_replica.cc:330] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: stopping tablet replica
I20260619 06:53:11.445218 25367 tablet_replica.cc:330] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: stopping tablet replica
I20260619 06:53:11.445290 25366 raft_consensus.cc:2229] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:11.445387 25367 raft_consensus.cc:2229] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:11.445554 25367 raft_consensus.cc:2258] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:11.445659 25366 raft_consensus.cc:2258] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:11.446552 25367 ts_tablet_manager.cc:1905] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:11.447016 25366 ts_tablet_manager.cc:1905] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:11.449183 25366 ts_tablet_manager.cc:1918] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:53:11.449276 25366 log.cc:1196] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/f94705fc8500438994d0785cf111baaf
I20260619 06:53:11.449611 25366 ts_tablet_manager.cc:1939] T f94705fc8500438994d0785cf111baaf P 8d3a04d80eb14efdb8be4de61c2236b8: Deleting consensus metadata
I20260619 06:53:11.450340 24697 catalog_manager.cc:4841] TS 8d3a04d80eb14efdb8be4de61c2236b8 (127.22.113.2:37043): tablet f94705fc8500438994d0785cf111baaf (table test-table [id=ef12579b583b434384a856151bcfec40]) successfully deleted
I20260619 06:53:11.451658 25367 ts_tablet_manager.cc:1918] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:53:11.451753 25367 log.cc:1196] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/f94705fc8500438994d0785cf111baaf
I20260619 06:53:11.452031 25367 ts_tablet_manager.cc:1939] T f94705fc8500438994d0785cf111baaf P 3162ac0d7b6945b69080a39f08ad88be: Deleting consensus metadata
I20260619 06:53:11.452773 24699 catalog_manager.cc:4841] TS 3162ac0d7b6945b69080a39f08ad88be (127.22.113.3:32965): tablet f94705fc8500438994d0785cf111baaf (table test-table [id=ef12579b583b434384a856151bcfec40]) successfully deleted
W20260619 06:53:11.453634 24696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:33561 (error 108)
I20260619 06:53:11.453858 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24924
I20260619 06:53:11.458343 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:11.457707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:33561 (local address 127.0.0.1:49708)
0619 06:53:11.458062 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:11.458177 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:33561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
W20260619 06:53:11.458556 24696 catalog_manager.cc:4584] TS e1864da7500a4c6b833b5c99561709be (127.22.113.1:33561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f94705fc8500438994d0785cf111baaf: Network error: Client connection negotiation failed: client connection to 127.22.113.1:33561: connect: Connection refused (error 111)
I20260619 06:53:11.462325 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25064
I20260619 06:53:11.471041 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 24680
2026-06-19T06:53:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6938 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:11 dist-test-slave-g8zg krb5kdc[25372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:11 dist-test-slave-g8zg krb5kdc[25372](info): set up 2 sockets
Jun 19 06:53:11 dist-test-slave-g8zg krb5kdc[25372](info): commencing operation
krb5kdc: starting...
W20260619 06:53:12.527199 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:12 dist-test-slave-g8zg krb5kdc[25372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:12.658586 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36999
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:36999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:12.783205 25385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:12.783681 25385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:12.783777 25385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:12.787978 25385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:12.788072 25385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:12.788111 25385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:12.788172 25385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:12.788210 25385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:12.793768 25385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:36999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:12.795109 25385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:12.796415 25385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:12.807008 25393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:12.807008 25391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:12.807036 25390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:12.807096 25385 server_base.cc:1034] running on GCE node
I20260619 06:53:12.808104 25385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:12.809463 25385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:12.810662 25385 hybrid_clock.cc:648] HybridClock initialized: now 1781851992810634 us; error 55 us; skew 500 ppm
Jun 19 06:53:12 dist-test-slave-g8zg krb5kdc[25372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:12.813789 25385 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:12.814870 25385 webserver.cc:460] Webserver started at http://127.22.113.62:37283/ using document root <none> and password file <none>
I20260619 06:53:12.815609 25385 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:12.815704 25385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:12.815970 25385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:12.818611 25385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "a9ff77e95b7247a3a22ba589fa6e35ec"
format_stamp: "Formatted at 2026-06-19 06:53:12 on dist-test-slave-g8zg"
server_key: "4eb89407c4d6f73b6a1a9de94266debe"
server_key_iv: "e1ea9dd4d4615e62e86b2dbd5fa3167b"
server_key_version: "encryptionkey@0"
I20260619 06:53:12.819566 25385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "a9ff77e95b7247a3a22ba589fa6e35ec"
format_stamp: "Formatted at 2026-06-19 06:53:12 on dist-test-slave-g8zg"
server_key: "4eb89407c4d6f73b6a1a9de94266debe"
server_key_iv: "e1ea9dd4d4615e62e86b2dbd5fa3167b"
server_key_version: "encryptionkey@0"
I20260619 06:53:12.823988 25385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:12.826805 25400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:12.828451 25385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:12.828611 25385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "a9ff77e95b7247a3a22ba589fa6e35ec"
format_stamp: "Formatted at 2026-06-19 06:53:12 on dist-test-slave-g8zg"
server_key: "4eb89407c4d6f73b6a1a9de94266debe"
server_key_iv: "e1ea9dd4d4615e62e86b2dbd5fa3167b"
server_key_version: "encryptionkey@0"
I20260619 06:53:12.828737 25385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:12.853680 25385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:12.857010 25385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:12.857234 25385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:12.865316 25385 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:36999
I20260619 06:53:12.865350 25452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:36999 every 8 connection(s)
I20260619 06:53:12.866320 25385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:12.869896 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:12.876632 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25385
I20260619 06:53:12.876748 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:12.876736 25453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: Bootstrap starting.
I20260619 06:53:12.877080 22980 external_mini_cluster.cc:1428] Setting key 6492be2deefcdd114030b7c3684cf494
I20260619 06:53:12.879904 25453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:12.880824 25453 log.cc:824] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:12.883389 25453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: No bootstrap required, opened a new log
Jun 19 06:53:12 dist-test-slave-g8zg krb5kdc[25372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:12.888304 25453 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } }
I20260619 06:53:12.888671 25453 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:12.888736 25453 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ff77e95b7247a3a22ba589fa6e35ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:12.889367 25453 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } }
I20260619 06:53:12.889605 25453 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:12.889698 25453 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:12.889860 25453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:12.891213 25453 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } }
I20260619 06:53:12.891688 25453 leader_election.cc:302] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ff77e95b7247a3a22ba589fa6e35ec; no voters:
I20260619 06:53:12.892127 25453 leader_election.cc:288] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:12.892203 25458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:12.892470 25458 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [term 1 LEADER]: Becoming Leader. State: Replica: a9ff77e95b7247a3a22ba589fa6e35ec, State: Running, Role: LEADER
I20260619 06:53:12.892894 25458 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } }
I20260619 06:53:12.893723 25453 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:12.894649 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:12.878719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.22.113.62:36999)
0619 06:53:12.879348 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:12.879377 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:12.879422 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:53:12.880327 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:12.880355 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:12.880750 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:12.881108 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:12.882232 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:12.883384 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:12.884283 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:12.884773 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:12.884952 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:12.888046 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:12.888051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:12.888067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:12.888093 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:12.890467 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:12.891133 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:12.891138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:12.891143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:12.891361 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:12.891665 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:12.891670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:12.891672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:12.892227 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:12.892456 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:12.893258 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 06:53:12.893561 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":169,"threads_started":1}
I20260619 06:53:12.895030 25460 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ff77e95b7247a3a22ba589fa6e35ec. Latest consensus state: current_term: 1 leader_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } } }
I20260619 06:53:12.895226 25460 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:12.895385 25459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ff77e95b7247a3a22ba589fa6e35ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36999 } } }
I20260619 06:53:12.895471 25459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec [sys.catalog]: This master's current role is: LEADER
W20260619 06:53:12.900256 25473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:12.900365 25473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:12.900444 25467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:12.901198 25467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:12.908259 25467 catalog_manager.cc:1347] Generated new cluster ID: 30db3ec4cca14b849cf4da1641744047
I20260619 06:53:12.908360 25467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:12.920629 25467 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:12.922214 25467 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:12.932758 25467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ff77e95b7247a3a22ba589fa6e35ec: Generated new TSK 0
I20260619 06:53:12.933714 25467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:12.970343 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:13.092485 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:13.092744 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:13.092788 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:13.097230 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:13.097294 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:13.097388 25479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:13.102691 25479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:13.103799 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:13.104887 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:13.115072 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:13.115083 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:13.115178 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:13.115324 25479 server_base.cc:1034] running on GCE node
I20260619 06:53:13.115867 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:13.116633 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:13.117869 25479 hybrid_clock.cc:648] HybridClock initialized: now 1781851993117826 us; error 51 us; skew 500 ppm
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:13.121155 25479 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:13.122246 25479 webserver.cc:460] Webserver started at http://127.22.113.1:35325/ using document root <none> and password file <none>
I20260619 06:53:13.122928 25479 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:13.123023 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:13.123343 25479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:13.125810 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "c1b4c7ef925c48fd9ede2259846d5c2c"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "57845ad88a9ae83e1e35354f02d0f52c"
server_key_iv: "0d337d9166dc33a32d8d0b73e1ea6f52"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.126451 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "c1b4c7ef925c48fd9ede2259846d5c2c"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "57845ad88a9ae83e1e35354f02d0f52c"
server_key_iv: "0d337d9166dc33a32d8d0b73e1ea6f52"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.130910 25479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:13.134102 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.135782 25479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:13.135926 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "c1b4c7ef925c48fd9ede2259846d5c2c"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "57845ad88a9ae83e1e35354f02d0f52c"
server_key_iv: "0d337d9166dc33a32d8d0b73e1ea6f52"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.136056 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:13.164906 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:13.168929 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:13.169200 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:13.169988 25479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:13.171054 25479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:13.171109 25479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.171154 25479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:13.171177 25479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.181874 25479 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:35533
I20260619 06:53:13.181964 25607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:35533 every 8 connection(s)
I20260619 06:53:13.182945 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:13.187666 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25479
I20260619 06:53:13.187795 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:13.188122 22980 external_mini_cluster.cc:1428] Setting key 7dae70f2a0b0c214341f1f6528fadf06
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:13.200099 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.185209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:42715 (local address 127.22.113.62:36999)
0619 06:53:13.185428 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:13.185432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:13.186388 (+ 956us) server_negotiation.cc:407] Connection header received
0619 06:53:13.188094 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:13.188098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:13.188158 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:13.188292 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:13.189845 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.190420 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.191713 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.191982 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.192102 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.195806 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:13.195810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:13.195813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:13.195839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:13.197497 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.198212 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.198215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.198216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.198300 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.198731 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.198733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.198734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.198946 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:13.199028 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:13.199821 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 06:53:13.199946 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 06:53:13.201004 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.185636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36999 (local address 127.22.113.1:42715)
0619 06:53:13.186183 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:13.186233 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:13.187802 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:13.188529 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:13.188538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:13.189044 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:13.189690 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.189709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.190561 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.190565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.191584 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.191594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.192141 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.192145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.192395 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.193175 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:13.193202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:13.195487 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:13.197647 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.197655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.197675 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.198093 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.198404 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.198409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.198411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.198615 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.199062 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:13.199069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:13.199497 (+ 428us) client_negotiation.cc:769] Sending connection context
0619 06:53:13.199726 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 06:53:13.200023 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:13.202577 25608 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36999
I20260619 06:53:13.202842 25608 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:13.203487 25608 heartbeater.cc:511] Master 127.22.113.62:36999 requested a full tablet report, sending...
I20260619 06:53:13.205435 25417 ts_manager.cc:194] Registered new tserver with Master: c1b4c7ef925c48fd9ede2259846d5c2c (127.22.113.1:35533)
I20260619 06:53:13.207327 25417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:42715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:13.222124 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:13.333078 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:13.333361 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:13.333411 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:13.337679 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:13.337746 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:13.337843 25613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:13.343353 25613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:13.344460 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:13.345480 25613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:13.355433 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:13.355435 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:13.355561 25613 server_base.cc:1034] running on GCE node
W20260619 06:53:13.355435 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:13.356272 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:13.356936 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:13.358110 25613 hybrid_clock.cc:648] HybridClock initialized: now 1781851993358086 us; error 39 us; skew 500 ppm
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:13.360925 25613 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:13.361883 25613 webserver.cc:460] Webserver started at http://127.22.113.2:41191/ using document root <none> and password file <none>
I20260619 06:53:13.362437 25613 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:13.362501 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:13.362727 25613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:13.364879 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "9654e21020bf4f5e8f1739c4b216c1fd"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "88a0ca546c573656a259fc3bdff368ff"
server_key_iv: "1325b21860f50b51ca368d5805507416"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.365406 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "9654e21020bf4f5e8f1739c4b216c1fd"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "88a0ca546c573656a259fc3bdff368ff"
server_key_iv: "1325b21860f50b51ca368d5805507416"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.369338 25613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:53:13.371946 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.373523 25613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:13.373704 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "9654e21020bf4f5e8f1739c4b216c1fd"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "88a0ca546c573656a259fc3bdff368ff"
server_key_iv: "1325b21860f50b51ca368d5805507416"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.373834 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:13.395174 25613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:13.398747 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:13.398964 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:13.399745 25613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:13.400698 25613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:13.400759 25613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.400806 25613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:13.400825 25613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.411209 25613 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:40017
I20260619 06:53:13.411288 25741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:40017 every 8 connection(s)
I20260619 06:53:13.412223 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:13.419994 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25613
I20260619 06:53:13.420151 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:13.420495 22980 external_mini_cluster.cc:1428] Setting key a28ae07e467d1c7c8873d611f5d942d5
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:13.427063 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.414423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:42901 (local address 127.22.113.62:36999)
0619 06:53:13.414624 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:13.414630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:13.415740 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 06:53:13.417100 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:13.417103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:13.417159 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:13.417280 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:13.418466 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.419051 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.420470 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.420806 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.420941 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.423436 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:13.423440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:13.423443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:13.423481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:13.424995 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.425512 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.425515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.425517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.425596 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.425942 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.425945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.425948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.426139 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:13.426247 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:13.426734 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 06:53:13.426867 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:13.427896 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.414867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36999 (local address 127.22.113.2:42901)
0619 06:53:13.415518 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:13.415621 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:13.416865 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:13.417455 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:13.417462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:13.417832 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:13.418320 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.418332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.419199 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.419202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.420286 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.420300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.420982 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.420986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.421179 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.421803 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:13.421826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:13.423218 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:13.425130 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.425135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.425150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.425403 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.425687 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.425691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.425693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.425840 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.426243 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:13.426249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:13.426496 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 06:53:13.426679 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 06:53:13.427029 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260619 06:53:13.429505 25742 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36999
I20260619 06:53:13.429769 25742 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:13.430289 25742 heartbeater.cc:511] Master 127.22.113.62:36999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:13.431540 25417 ts_manager.cc:194] Registered new tserver with Master: 9654e21020bf4f5e8f1739c4b216c1fd (127.22.113.2:40017)
I20260619 06:53:13.432417 25417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:42901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:13.455092 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:13.566562 25747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:13.566869 25747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:13.566947 25747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:13.570968 25747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:13.571074 25747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:13.571166 25747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:13.576496 25747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:13.577625 25747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:13.578706 25747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:13.590631 25753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:13.590621 25755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:13.590732 25752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:13.590814 25747 server_base.cc:1034] running on GCE node
I20260619 06:53:13.591681 25747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:13.592519 25747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:13.593737 25747 hybrid_clock.cc:648] HybridClock initialized: now 1781851993593690 us; error 56 us; skew 500 ppm
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:13.597575 25747 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:13.598728 25747 webserver.cc:460] Webserver started at http://127.22.113.3:38409/ using document root <none> and password file <none>
I20260619 06:53:13.599502 25747 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:13.599603 25747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:13.599922 25747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:13.602600 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "cdcbbb93bf884c8db0b9a3135099f8da"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "c5844bb790481b06d9b81b077596412b"
server_key_iv: "3ef83233130f14c5ff1c0bf486cb4ea7"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.603441 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "cdcbbb93bf884c8db0b9a3135099f8da"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "c5844bb790481b06d9b81b077596412b"
server_key_iv: "3ef83233130f14c5ff1c0bf486cb4ea7"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.608104 25747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:13.611364 25762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.612972 25747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:13.613158 25747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "cdcbbb93bf884c8db0b9a3135099f8da"
format_stamp: "Formatted at 2026-06-19 06:53:13 on dist-test-slave-g8zg"
server_key: "c5844bb790481b06d9b81b077596412b"
server_key_iv: "3ef83233130f14c5ff1c0bf486cb4ea7"
server_key_version: "encryptionkey@0"
I20260619 06:53:13.613301 25747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:13.629580 25747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:13.633214 25747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:13.633443 25747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:13.634155 25747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:13.635076 25747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:13.635128 25747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.635170 25747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:13.635186 25747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:13.646391 25747 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:34449
I20260619 06:53:13.646415 25875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:34449 every 8 connection(s)
I20260619 06:53:13.647540 25747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:13.653016 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25747
I20260619 06:53:13.653139 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:13.653522 22980 external_mini_cluster.cc:1428] Setting key efae619dba62312cf392312d5fbc6b01
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:13.662900 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.649785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:45857 (local address 127.22.113.62:36999)
0619 06:53:13.650140 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:13.650144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:13.651068 (+ 924us) server_negotiation.cc:407] Connection header received
0619 06:53:13.652549 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:13.652552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:13.652612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:13.652715 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:13.654156 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.654934 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.656296 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.656514 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.656602 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.659152 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:13.659155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:13.659158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:13.659183 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:13.660856 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.661403 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.661406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.661407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.661496 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:13.661830 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:13.661832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:13.661833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:13.662039 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:13.662122 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:13.662635 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 06:53:13.662756 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260619 06:53:13.663993 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:13.650265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36999 (local address 127.22.113.3:45857)
0619 06:53:13.650869 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:13.650922 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:13.652313 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:13.652897 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:13.652904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:13.653430 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:13.653975 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.653994 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.655075 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.655078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.656186 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:13.656196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:13.656650 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:13.656652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:13.656795 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:13.657427 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:13.657443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:13.658985 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:13.660989 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.660994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.661007 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.661279 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.661597 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:13.661599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:13.661601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:13.661735 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:13.662137 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:13.662142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:13.662403 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 06:53:13.662590 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 06:53:13.662869 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":166,"threads_started":1}
I20260619 06:53:13.665575 25876 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36999
I20260619 06:53:13.665871 25876 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:13.666404 25876 heartbeater.cc:511] Master 127.22.113.62:36999 requested a full tablet report, sending...
I20260619 06:53:13.667845 25417 ts_manager.cc:194] Registered new tserver with Master: cdcbbb93bf884c8db0b9a3135099f8da (127.22.113.3:34449)
I20260619 06:53:13.668448 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:53:13.668605 25417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:45857
W20260619 06:53:13.684369 25456 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:13.682684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.22.113.62:36999)
0619 06:53:13.682835 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:13.682839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:13.683050 (+ 211us) server_negotiation.cc:407] Connection header received
0619 06:53:13.683179 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:13.683182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:13.683232 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:13.683387 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:13.684177 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48086: BlockingRecv error: recv got EOF from 127.0.0.1:48086 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:53:13.686017 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25479
I20260619 06:53:13.692435 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25613
I20260619 06:53:13.698803 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25747
I20260619 06:53:13.704965 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25385
2026-06-19T06:53:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2232 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25888](info): set up 2 sockets
Jun 19 06:53:13 dist-test-slave-g8zg krb5kdc[25888](info): commencing operation
krb5kdc: starting...
W20260619 06:53:14.755453 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:14 dist-test-slave-g8zg krb5kdc[25888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:14.891443 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:44055
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:14.999979 25901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:15.000239 25901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:15.000288 25901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:15.004261 25901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:15.004317 25901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:15.004338 25901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:15.004354 25901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:15.004369 25901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:15.009511 25901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:15.010460 25901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:15.011435 25901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:15.020272 25907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:15.020270 25906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:15.020315 25909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:15.020784 25901 server_base.cc:1034] running on GCE node
I20260619 06:53:15.021289 25901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:15.022348 25901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:15.023610 25901 hybrid_clock.cc:648] HybridClock initialized: now 1781851995023589 us; error 44 us; skew 500 ppm
Jun 19 06:53:15 dist-test-slave-g8zg krb5kdc[25888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851995, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:15.032097 25901 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:15.033208 25901 webserver.cc:460] Webserver started at http://127.22.113.62:45979/ using document root <none> and password file <none>
I20260619 06:53:15.033864 25901 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:15.033964 25901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:15.034214 25901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:15.036571 25901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "065bbd0731cb4875b830fc9bb3018f91"
format_stamp: "Formatted at 2026-06-19 06:53:15 on dist-test-slave-g8zg"
server_key: "246473be782142ac3e10f56b3492bfcf"
server_key_iv: "dfed933402b7583772f6d05a10664db5"
server_key_version: "encryptionkey@0"
I20260619 06:53:15.037182 25901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "065bbd0731cb4875b830fc9bb3018f91"
format_stamp: "Formatted at 2026-06-19 06:53:15 on dist-test-slave-g8zg"
server_key: "246473be782142ac3e10f56b3492bfcf"
server_key_iv: "dfed933402b7583772f6d05a10664db5"
server_key_version: "encryptionkey@0"
I20260619 06:53:15.041541 25901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:15.044456 25916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:15.045992 25901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:53:15.046151 25901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "065bbd0731cb4875b830fc9bb3018f91"
format_stamp: "Formatted at 2026-06-19 06:53:15 on dist-test-slave-g8zg"
server_key: "246473be782142ac3e10f56b3492bfcf"
server_key_iv: "dfed933402b7583772f6d05a10664db5"
server_key_version: "encryptionkey@0"
I20260619 06:53:15.046262 25901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:15.055305 25901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:15.058143 25901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:15.058478 25901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:15.066171 25901 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:44055
I20260619 06:53:15.066577 25968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:44055 every 8 connection(s)
I20260619 06:53:15.067165 25901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:15.068704 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 25901
I20260619 06:53:15.068862 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:15.069154 22980 external_mini_cluster.cc:1428] Setting key 0e4e5994520b6886143adf411eb895e5
I20260619 06:53:15.071359 25969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:53:15 dist-test-slave-g8zg krb5kdc[25888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:15.078714 25969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91: Bootstrap starting.
I20260619 06:53:15.081633 25969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:15.082506 25969 log.cc:824] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:15.084520 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:15.070751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.22.113.62:44055)
0619 06:53:15.071395 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:15.071417 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:15.071486 (+ 69us) server_negotiation.cc:407] Connection header received
0619 06:53:15.072160 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:15.072177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:15.072543 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:15.072891 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:15.073743 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:15.074767 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:15.075683 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:15.076171 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:15.076362 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:15.079518 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:15.079523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:15.079537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:15.079565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:15.081247 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:15.081806 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:15.081811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:15.081815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:15.081926 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:15.082201 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:15.082205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:15.082207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:15.082679 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:15.082869 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:15.083408 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 06:53:15.083702 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":186,"threads_started":1}
I20260619 06:53:15.085172 25969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91: No bootstrap required, opened a new log
I20260619 06:53:15.088125 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:53:15.089946 25969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } }
I20260619 06:53:15.090241 25969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:15.090302 25969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065bbd0731cb4875b830fc9bb3018f91, State: Initialized, Role: FOLLOWER
I20260619 06:53:15.090762 25969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } }
I20260619 06:53:15.090920 25969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:15.091008 25969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:15.091121 25969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:15.092797 25969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } }
I20260619 06:53:15.093333 25969 leader_election.cc:302] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065bbd0731cb4875b830fc9bb3018f91; no voters:
I20260619 06:53:15.093760 25969 leader_election.cc:288] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:15.093947 25975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:15.094242 25975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [term 1 LEADER]: Becoming Leader. State: Replica: 065bbd0731cb4875b830fc9bb3018f91, State: Running, Role: LEADER
I20260619 06:53:15.094671 25975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } }
I20260619 06:53:15.095551 25969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:15.096812 25977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065bbd0731cb4875b830fc9bb3018f91. Latest consensus state: current_term: 1 leader_uuid: "065bbd0731cb4875b830fc9bb3018f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } } }
I20260619 06:53:15.096856 25976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "065bbd0731cb4875b830fc9bb3018f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065bbd0731cb4875b830fc9bb3018f91" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 44055 } } }
I20260619 06:53:15.097086 25977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:15.097100 25976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:15.097577 25985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:15.102202 25985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:15.108100 25985 catalog_manager.cc:1347] Generated new cluster ID: b2c13aa5de8a458abede80dac7529aaf
I20260619 06:53:15.108160 25985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:15.111490 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:15.104486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.22.113.62:44055)
0619 06:53:15.104598 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:15.104601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:15.104980 (+ 379us) server_negotiation.cc:407] Connection header received
0619 06:53:15.105129 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:15.105133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:15.105204 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:15.105314 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:15.105979 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:15.107394 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:15.108773 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:15.109188 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:15.109306 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:15.110442 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:15.110446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:15.110449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:15.110955 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:15.110989 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:15.111007 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:15.111098 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:15.111287 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 06:53:15.111353 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 06:53:15.121416 25985 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:15.122349 25985 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:15.131043 25985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 065bbd0731cb4875b830fc9bb3018f91: Generated new TSK 0
I20260619 06:53:15.132228 25985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:15.144227 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 25901
2026-06-19T06:53:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1436 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:15 dist-test-slave-g8zg krb5kdc[26002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:15 dist-test-slave-g8zg krb5kdc[26002](info): set up 2 sockets
Jun 19 06:53:15 dist-test-slave-g8zg krb5kdc[26002](info): commencing operation
krb5kdc: starting...
W20260619 06:53:16.207422 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:16 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:16.344836 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39565
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:39565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:16.457122 26015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:16.457417 26015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:16.457463 26015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:16.461457 26015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:16.461524 26015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:16.461547 26015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:16.461568 26015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:16.461632 26015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:16.466943 26015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:39565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:16.468140 26015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:16.469226 26015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:16.478430 26021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:16.478463 26020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:16.478459 26023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:16.478854 26015 server_base.cc:1034] running on GCE node
I20260619 06:53:16.479621 26015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:16.480981 26015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:16.482167 26015 hybrid_clock.cc:648] HybridClock initialized: now 1781851996482132 us; error 53 us; skew 500 ppm
Jun 19 06:53:16 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851996, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:16.485486 26015 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:16.486510 26015 webserver.cc:460] Webserver started at http://127.22.113.62:46379/ using document root <none> and password file <none>
I20260619 06:53:16.487172 26015 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:16.487296 26015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:16.487648 26015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:16.490764 26015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "1334039155084813a7e32139fc84c05a"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "9daebbd9e4c37907c0d2cd63ceb6ec0b"
server_key_iv: "84dd50e345617c1cab5e85f666151334"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.491506 26015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "1334039155084813a7e32139fc84c05a"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "9daebbd9e4c37907c0d2cd63ceb6ec0b"
server_key_iv: "84dd50e345617c1cab5e85f666151334"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.496054 26015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:16.498908 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:16.500514 26015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:53:16.500666 26015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "1334039155084813a7e32139fc84c05a"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "9daebbd9e4c37907c0d2cd63ceb6ec0b"
server_key_iv: "84dd50e345617c1cab5e85f666151334"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.500797 26015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:16.519775 26015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:16.523103 26015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:16.523346 26015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:16.531759 26015 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:39565
I20260619 06:53:16.532106 26082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:39565 every 8 connection(s)
I20260619 06:53:16.532918 26015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:16.536185 26083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:16.541815 26083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a: Bootstrap starting.
I20260619 06:53:16.543136 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26015
I20260619 06:53:16.543315 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:16.543649 22980 external_mini_cluster.cc:1428] Setting key b78491f3cee9532deaf8e749e49cc621
I20260619 06:53:16.544611 26083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:16.545435 26083 log.cc:824] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:16.547878 26083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a: No bootstrap required, opened a new log
I20260619 06:53:16.552266 26083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } }
I20260619 06:53:16.552595 26083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:16.552667 26083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1334039155084813a7e32139fc84c05a, State: Initialized, Role: FOLLOWER
I20260619 06:53:16.553249 26083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } }
I20260619 06:53:16.553426 26083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:16.553515 26083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:16.553642 26083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:53:16 dist-test-slave-g8zg krb5kdc[26002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:16.554726 26083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } }
I20260619 06:53:16.555085 26083 leader_election.cc:302] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1334039155084813a7e32139fc84c05a; no voters:
I20260619 06:53:16.555492 26083 leader_election.cc:288] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:16.555608 26088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:16.555823 26088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [term 1 LEADER]: Becoming Leader. State: Replica: 1334039155084813a7e32139fc84c05a, State: Running, Role: LEADER
I20260619 06:53:16.556136 26088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } }
I20260619 06:53:16.556813 26083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:16.557372 26089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1334039155084813a7e32139fc84c05a. Latest consensus state: current_term: 1 leader_uuid: "1334039155084813a7e32139fc84c05a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } } }
I20260619 06:53:16.557869 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1334039155084813a7e32139fc84c05a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1334039155084813a7e32139fc84c05a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39565 } } }
I20260619 06:53:16.558022 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:16.558459 26097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:16.557546 26089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:16.561048 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:16.545051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.22.113.62:39565)
0619 06:53:16.545617 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:16.545642 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:16.545683 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:53:16.546667 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:16.546692 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:16.547086 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:16.547477 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:16.548585 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.549667 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.551026 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.551616 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.551753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:16.554914 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:16.554923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:16.554944 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:16.554984 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:16.557110 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:16.557638 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:16.557647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:16.557654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:16.557838 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:16.558183 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:16.558186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:16.558188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:16.558779 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:16.558973 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:16.559590 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 06:53:16.559951 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":217,"threads_started":1}
I20260619 06:53:16.563537 26097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:16.570115 26097 catalog_manager.cc:1347] Generated new cluster ID: 36e49fe8355d40f5ba1c403a7423fc62
I20260619 06:53:16.570211 26097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:16.589085 26097 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:16.590427 26097 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:16.600718 26097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1334039155084813a7e32139fc84c05a: Generated new TSK 0
I20260619 06:53:16.601543 26097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:16.635468 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:16.759195 26109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:16.759584 26109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:16.759667 26109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:16.764014 26109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:16.764117 26109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:16.764230 26109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:16.769970 26109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:16.771400 26109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:16.772666 26109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:16.783746 26114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:16.783805 26115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:16.783763 26117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:16.784211 26109 server_base.cc:1034] running on GCE node
I20260619 06:53:16.784720 26109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:16.785549 26109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:16.786942 26109 hybrid_clock.cc:648] HybridClock initialized: now 1781851996786911 us; error 40 us; skew 500 ppm
Jun 19 06:53:16 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851996, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:16.790457 26109 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:16.791707 26109 webserver.cc:460] Webserver started at http://127.22.113.1:32927/ using document root <none> and password file <none>
I20260619 06:53:16.792416 26109 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:16.792487 26109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:16.792718 26109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:16.795480 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "f02b3e77a0b240d5b3f46c351f18c362"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "2af1a4e3759d5b064ad32e196cea9deb"
server_key_iv: "2fceaff2d9d3c4eac0a59606e71da105"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.796165 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "f02b3e77a0b240d5b3f46c351f18c362"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "2af1a4e3759d5b064ad32e196cea9deb"
server_key_iv: "2fceaff2d9d3c4eac0a59606e71da105"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.800988 26109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:53:16.804275 26124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:16.806066 26109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:16.806257 26109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "f02b3e77a0b240d5b3f46c351f18c362"
format_stamp: "Formatted at 2026-06-19 06:53:16 on dist-test-slave-g8zg"
server_key: "2af1a4e3759d5b064ad32e196cea9deb"
server_key_iv: "2fceaff2d9d3c4eac0a59606e71da105"
server_key_version: "encryptionkey@0"
I20260619 06:53:16.806428 26109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:16.817857 26109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:16.821694 26109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:16.821960 26109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:16.822742 26109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:16.823897 26109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:16.823953 26109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:16.824034 26109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:16.824071 26109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:16.835989 26109 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:40183
I20260619 06:53:16.836020 26237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:40183 every 8 connection(s)
I20260619 06:53:16.837136 26109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:16.843477 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26109
I20260619 06:53:16.843638 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:16.843966 22980 external_mini_cluster.cc:1428] Setting key 00db8ec95fb7712c60f9043346c0b7c1
Jun 19 06:53:16 dist-test-slave-g8zg krb5kdc[26002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851996, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:16.855082 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:16.839476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:43051 (local address 127.22.113.62:39565)
0619 06:53:16.839635 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:16.839639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:16.840792 (+ 1153us) server_negotiation.cc:407] Connection header received
0619 06:53:16.842265 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:16.842268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:16.842335 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:16.842442 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:16.844079 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.844704 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.846170 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.846457 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.846565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:16.850118 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:16.850125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:16.850132 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:16.850174 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:16.852319 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:16.853058 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:16.853065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:16.853071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:16.853216 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:16.853627 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:16.853631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:16.853634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:16.853933 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:16.854072 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:16.854650 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 06:53:16.854820 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:53:16.855758 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:16.839995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39565 (local address 127.22.113.1:43051)
0619 06:53:16.840584 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:16.840636 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:16.841983 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:16.842648 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:16.842656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:16.843074 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:16.843826 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:16.843857 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.844855 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.844858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:16.846005 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:16.846019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:16.846641 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:16.846648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:16.846900 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:16.847711 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:16.847738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:16.849915 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:16.852620 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:16.852625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:16.852637 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:16.852910 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:16.853333 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:16.853336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:16.853337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:16.853495 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:16.854069 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:16.854073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:16.854322 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 06:53:16.854515 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 06:53:16.854776 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:16.857280 26238 heartbeater.cc:347] Connected to a master server at 127.22.113.62:39565
I20260619 06:53:16.857721 26238 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:16.858517 26238 heartbeater.cc:511] Master 127.22.113.62:39565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:16.860613 26045 ts_manager.cc:194] Registered new tserver with Master: f02b3e77a0b240d5b3f46c351f18c362 (127.22.113.1:40183)
I20260619 06:53:16.861996 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:43051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:16.878242 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:16.993415 26243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:16.993717 26243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:16.993788 26243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:16.997849 26243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:16.997951 26243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:16.998036 26243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:17.003495 26243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:17.004688 26243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:17.005788 26243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:17.016482 26251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:17.016470 26248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:17.016501 26249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:17.016592 26243 server_base.cc:1034] running on GCE node
I20260619 06:53:17.017362 26243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:17.018095 26243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:17.019361 26243 hybrid_clock.cc:648] HybridClock initialized: now 1781851997019309 us; error 60 us; skew 500 ppm
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:17.022372 26243 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:17.023422 26243 webserver.cc:460] Webserver started at http://127.22.113.2:36499/ using document root <none> and password file <none>
I20260619 06:53:17.024034 26243 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:17.024130 26243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:17.024366 26243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:17.026636 26243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "02208ffd034e435684fa51220cc5c8ff"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "30180c8717cd0461f88423b884e8dc4f"
server_key_iv: "60d7bf8a18ad18180e811b9c03ec9104"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.027187 26243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "02208ffd034e435684fa51220cc5c8ff"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "30180c8717cd0461f88423b884e8dc4f"
server_key_iv: "60d7bf8a18ad18180e811b9c03ec9104"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.031313 26243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:17.034132 26258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.035733 26243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:17.035890 26243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "02208ffd034e435684fa51220cc5c8ff"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "30180c8717cd0461f88423b884e8dc4f"
server_key_iv: "60d7bf8a18ad18180e811b9c03ec9104"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.036011 26243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:17.053030 26243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:17.056733 26243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:17.056959 26243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:17.057750 26243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:17.058764 26243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:17.058844 26243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.058921 26243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:17.058960 26243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.070242 26243 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:34221
I20260619 06:53:17.070308 26371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:34221 every 8 connection(s)
I20260619 06:53:17.071354 26243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:17.075526 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26243
I20260619 06:53:17.075685 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:17.075986 22980 external_mini_cluster.cc:1428] Setting key 1a3226ad3de72e4bd2ae0992aec2f665
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:17.089246 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:17.073709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57811 (local address 127.22.113.62:39565)
0619 06:53:17.073894 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:17.073898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:17.074846 (+ 948us) server_negotiation.cc:407] Connection header received
0619 06:53:17.076674 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:17.076678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:17.076740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:17.076862 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:17.078524 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.079451 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.081373 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.081594 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.081682 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:17.084949 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:17.084955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:17.084961 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:17.084998 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:17.086819 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.087458 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.087464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.087469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.087567 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.087937 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.087939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.087940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.088189 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:17.088297 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:17.088971 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 06:53:17.089098 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:53:17.090078 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:17.074077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39565 (local address 127.22.113.2:57811)
0619 06:53:17.074646 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:17.074698 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:17.076317 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:17.077094 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:17.077104 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:17.077708 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:17.078332 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:17.078346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.079646 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.079653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:17.081233 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:17.081247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.081776 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.081782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:17.082028 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:17.082748 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:17.082775 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:17.084737 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:17.086972 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:17.086977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:17.086990 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:17.087347 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:17.087670 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:17.087673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:17.087674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:17.087827 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:17.088301 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:17.088306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:17.088666 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 06:53:17.088838 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:53:17.089133 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:53:17.091431 26372 heartbeater.cc:347] Connected to a master server at 127.22.113.62:39565
I20260619 06:53:17.091742 26372 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:17.092377 26372 heartbeater.cc:511] Master 127.22.113.62:39565 requested a full tablet report, sending...
I20260619 06:53:17.093719 26045 ts_manager.cc:194] Registered new tserver with Master: 02208ffd034e435684fa51220cc5c8ff (127.22.113.2:34221)
I20260619 06:53:17.094339 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:57811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:17.106948 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:17.231036 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:17.231356 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:17.231400 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:17.235817 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:17.235888 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:17.235951 26377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:17.241634 26377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:17.242875 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:17.244148 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:17.255000 26385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:17.255000 26383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:17.255092 26382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:17.255482 26377 server_base.cc:1034] running on GCE node
I20260619 06:53:17.256018 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:17.256779 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:17.257970 26377 hybrid_clock.cc:648] HybridClock initialized: now 1781851997257931 us; error 53 us; skew 500 ppm
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:17.260970 26377 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:17.261898 26377 webserver.cc:460] Webserver started at http://127.22.113.3:34215/ using document root <none> and password file <none>
I20260619 06:53:17.262486 26377 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:17.262550 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:17.262724 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:17.264884 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "c068f571b51d47b897b00b7a746e16af"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "207972554082aba5f79338d7d6e952f7"
server_key_iv: "6faf552b4c6a8252121bf717d80cc964"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.265511 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "c068f571b51d47b897b00b7a746e16af"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "207972554082aba5f79338d7d6e952f7"
server_key_iv: "6faf552b4c6a8252121bf717d80cc964"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.269508 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:17.272194 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.273810 26377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:17.273976 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "c068f571b51d47b897b00b7a746e16af"
format_stamp: "Formatted at 2026-06-19 06:53:17 on dist-test-slave-g8zg"
server_key: "207972554082aba5f79338d7d6e952f7"
server_key_iv: "6faf552b4c6a8252121bf717d80cc964"
server_key_version: "encryptionkey@0"
I20260619 06:53:17.274102 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:17.284636 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:17.287878 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:17.288079 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:17.288664 26377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:17.289510 26377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:17.289579 26377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.289650 26377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:17.289685 26377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:17.299751 26377 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40649
I20260619 06:53:17.299870 26505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40649 every 8 connection(s)
I20260619 06:53:17.300797 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:17.304838 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26377
I20260619 06:53:17.304973 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:17.305274 22980 external_mini_cluster.cc:1428] Setting key 0a53587f6aa8818fddb912fdfcc378dd
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:17.316021 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:17.303072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:33959 (local address 127.22.113.62:39565)
0619 06:53:17.303301 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:17.303307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:17.304279 (+ 972us) server_negotiation.cc:407] Connection header received
0619 06:53:17.305985 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:17.305989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:17.306052 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:17.306142 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:17.307502 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.308084 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.309291 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.309515 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.309627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:17.312315 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:17.312319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:17.312322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:17.312354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:17.313836 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.314338 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.314341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.314342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.314417 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.314732 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.314734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.314735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.314930 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:17.315074 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:17.315756 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 06:53:17.315877 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:53:17.316956 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:17.303540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39565 (local address 127.22.113.3:33959)
0619 06:53:17.304080 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:17.304132 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:17.305752 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:17.306350 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:17.306357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:17.306756 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:17.307363 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:17.307379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.308212 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.308216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:17.309149 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:17.309158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.309673 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.309677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:17.309853 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:17.310477 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:17.310494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:17.312146 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:17.313948 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:17.313953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:17.313965 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:17.314223 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:17.314500 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:17.314502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:17.314503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:17.314639 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:17.315091 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:17.315102 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:17.315458 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 06:53:17.315642 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:53:17.315941 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":124,"threads_started":1}
I20260619 06:53:17.318463 26506 heartbeater.cc:347] Connected to a master server at 127.22.113.62:39565
I20260619 06:53:17.318794 26506 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:17.319528 26506 heartbeater.cc:511] Master 127.22.113.62:39565 requested a full tablet report, sending...
I20260619 06:53:17.320665 26045 ts_manager.cc:194] Registered new tserver with Master: c068f571b51d47b897b00b7a746e16af (127.22.113.3:40649)
I20260619 06:53:17.321264 26045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:33959
I20260619 06:53:17.331295 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851997, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:17.364282 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:17.350854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.22.113.62:39565)
0619 06:53:17.351059 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:17.351063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:17.351177 (+ 114us) server_negotiation.cc:407] Connection header received
0619 06:53:17.351416 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:17.351419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:17.351475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:17.351558 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:17.352330 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.352893 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.353850 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:17.354097 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:17.354205 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:17.360859 (+ 6654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:17.360864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:17.360866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:17.360902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:17.362348 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.362825 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.362828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.362829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.362931 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:17.363217 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:17.363219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:17.363220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:17.363554 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:17.363698 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:17.363934 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:53:17.364060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 06:53:17.364852 26045 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37206
I20260619 06:53:17.365814 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26109
I20260619 06:53:17.374162 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26243
I20260619 06:53:17.380769 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26377
I20260619 06:53:17.386811 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26015
2026-06-19T06:53:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2244 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26518](info): set up 2 sockets
Jun 19 06:53:17 dist-test-slave-g8zg krb5kdc[26518](info): commencing operation
krb5kdc: starting...
W20260619 06:53:18.495648 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:18 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:18.634096 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36829
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:18.749364 26531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:18.749691 26531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:18.749733 26531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:18.753994 26531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:18.754065 26531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:18.754084 26531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:18.754101 26531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:18.754118 26531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:18.759610 26531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:36829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:18.760746 26531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:18.761796 26531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:18.771165 26536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:18.771162 26537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:18.771272 26539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:18.771641 26531 server_base.cc:1034] running on GCE node
I20260619 06:53:18.772406 26531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:18.773778 26531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:18.774979 26531 hybrid_clock.cc:648] HybridClock initialized: now 1781851998774940 us; error 56 us; skew 500 ppm
Jun 19 06:53:18 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851998, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:18.778153 26531 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:18.779346 26531 webserver.cc:460] Webserver started at http://127.22.113.62:39611/ using document root <none> and password file <none>
I20260619 06:53:18.780040 26531 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:18.780107 26531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:18.780318 26531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:18.782658 26531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "9ec9323b365d470abc554c42bf371874"
format_stamp: "Formatted at 2026-06-19 06:53:18 on dist-test-slave-g8zg"
server_key: "a29a46c64e1c4ae62031d6fb6ece7815"
server_key_iv: "6f9a4494c2557e5421876a067783d7cc"
server_key_version: "encryptionkey@0"
I20260619 06:53:18.783233 26531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "9ec9323b365d470abc554c42bf371874"
format_stamp: "Formatted at 2026-06-19 06:53:18 on dist-test-slave-g8zg"
server_key: "a29a46c64e1c4ae62031d6fb6ece7815"
server_key_iv: "6f9a4494c2557e5421876a067783d7cc"
server_key_version: "encryptionkey@0"
I20260619 06:53:18.787633 26531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:18.790473 26546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:18.792042 26531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:18.792205 26531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "9ec9323b365d470abc554c42bf371874"
format_stamp: "Formatted at 2026-06-19 06:53:18 on dist-test-slave-g8zg"
server_key: "a29a46c64e1c4ae62031d6fb6ece7815"
server_key_iv: "6f9a4494c2557e5421876a067783d7cc"
server_key_version: "encryptionkey@0"
I20260619 06:53:18.792364 26531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:18.806540 26531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:18.810138 26531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:18.810364 26531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:18.818890 26531 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:36829
I20260619 06:53:18.818923 26598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:36829 every 8 connection(s)
I20260619 06:53:18.820091 26531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:18.820825 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26531
I20260619 06:53:18.820935 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:18.821277 22980 external_mini_cluster.cc:1428] Setting key 88b06cec643660cc0a1bfcd144e4523f
I20260619 06:53:18.823663 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:18.830447 26599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: Bootstrap starting.
Jun 19 06:53:18 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:18.833039 26599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:18.833969 26599 log.cc:824] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:18.836304 26599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: No bootstrap required, opened a new log
I20260619 06:53:18.837695 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:18.822998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.22.113.62:36829)
0619 06:53:18.823514 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:18.823537 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:18.823585 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:53:18.824483 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:18.824505 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:18.824907 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:18.825277 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:18.826028 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:18.827545 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:18.828437 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:18.829125 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:18.829301 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:18.832171 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:18.832177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:18.832192 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:18.832224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:18.834389 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:18.834958 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:18.834964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:18.834971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:18.835105 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:18.835437 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:18.835439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:18.835440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:18.835930 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:18.836098 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:18.836485 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 06:53:18.836800 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":191,"threads_started":1}
I20260619 06:53:18.840350 26599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } }
I20260619 06:53:18.840608 26599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:18.840691 26599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec9323b365d470abc554c42bf371874, State: Initialized, Role: FOLLOWER
I20260619 06:53:18.841096 26599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } }
I20260619 06:53:18.841215 26599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:18.841308 26599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:18.841435 26599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:18.842332 26599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } }
I20260619 06:53:18.842682 26599 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec9323b365d470abc554c42bf371874; no voters:
I20260619 06:53:18.843005 26599 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:18.843230 26604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:18.843659 26604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec9323b365d470abc554c42bf371874, State: Running, Role: LEADER
I20260619 06:53:18.844069 26604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } }
I20260619 06:53:18.844416 26599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:18.846354 26606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec9323b365d470abc554c42bf371874. Latest consensus state: current_term: 1 leader_uuid: "9ec9323b365d470abc554c42bf371874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } } }
I20260619 06:53:18.846503 26606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:18.846818 26605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec9323b365d470abc554c42bf371874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec9323b365d470abc554c42bf371874" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36829 } } }
I20260619 06:53:18.847184 26605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874 [sys.catalog]: This master's current role is: LEADER
W20260619 06:53:18.849452 26618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:18.849543 26618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:18.849653 26619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:18.850386 26619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:18.855902 26619 catalog_manager.cc:1347] Generated new cluster ID: 4df029aaa7624bbf91c40308732ec568
I20260619 06:53:18.855974 26619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:18.870096 26619 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:18.871066 26619 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:18.881700 26619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec9323b365d470abc554c42bf371874: Generated new TSK 0
I20260619 06:53:18.882400 26619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:18.919034 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:19.040580 26625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:19.040884 26625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:19.040932 26625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:19.045409 26625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:19.045472 26625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:19.045589 26625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:19.051229 26625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:19.052552 26625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:19.053772 26625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:19.064851 26630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.064894 26633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.064988 26631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:19.065340 26625 server_base.cc:1034] running on GCE node
I20260619 06:53:19.065797 26625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:19.066526 26625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:19.067718 26625 hybrid_clock.cc:648] HybridClock initialized: now 1781851999067701 us; error 40 us; skew 500 ppm
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:19.070760 26625 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:19.071888 26625 webserver.cc:460] Webserver started at http://127.22.113.1:33107/ using document root <none> and password file <none>
I20260619 06:53:19.072552 26625 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:19.072643 26625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:19.072904 26625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:19.075397 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "0c69ff246db5434db50113fc59c536c5"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "21796232744c5d0501a7ea6dc9b0edea"
server_key_iv: "1747d8dececcc449201855a60d4ac9ad"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.076035 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "0c69ff246db5434db50113fc59c536c5"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "21796232744c5d0501a7ea6dc9b0edea"
server_key_iv: "1747d8dececcc449201855a60d4ac9ad"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.080807 26625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:19.083774 26640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.085285 26625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:19.085443 26625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "0c69ff246db5434db50113fc59c536c5"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "21796232744c5d0501a7ea6dc9b0edea"
server_key_iv: "1747d8dececcc449201855a60d4ac9ad"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.085569 26625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:19.119220 26625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:19.122658 26625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:19.122892 26625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:19.123646 26625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:19.124806 26625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:19.124888 26625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.124958 26625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:19.124982 26625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.136193 26625 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:45025
I20260619 06:53:19.136257 26753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:45025 every 8 connection(s)
I20260619 06:53:19.137333 26625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:19.146441 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26625
I20260619 06:53:19.146632 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:19.147122 22980 external_mini_cluster.cc:1428] Setting key 0b5348185e66772f2b8dc047e39ac7c0
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:19.154054 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.139648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44085 (local address 127.22.113.62:36829)
0619 06:53:19.139838 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:19.139843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:19.140839 (+ 996us) server_negotiation.cc:407] Connection header received
0619 06:53:19.142391 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:19.142394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:19.142453 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:19.142562 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:19.143885 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.144506 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.145789 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.146052 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.146164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.149599 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:19.149603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:19.149605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:19.149636 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:19.151454 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.152221 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.152223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.152224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.152297 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.152735 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.152738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.152739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.152904 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:19.152992 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:19.153806 (+ 814us) server_negotiation.cc:299] Negotiation successful
0619 06:53:19.153914 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:53:19.155737 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.140093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36829 (local address 127.22.113.1:44085)
0619 06:53:19.140653 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:19.140704 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:19.142160 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:19.142750 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:19.142758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:19.143157 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:19.143734 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.143752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.144639 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.144643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.145672 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.145682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.146225 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.146229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.146391 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.147194 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:19.147219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:19.149386 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:19.151589 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.151600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.151622 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.152091 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.152397 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.152401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.152403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.152623 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.153028 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:19.153038 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:19.153488 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 06:53:19.153751 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 06:53:19.154206 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":118,"threads_started":1}
I20260619 06:53:19.157414 26754 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36829
I20260619 06:53:19.157768 26754 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:19.158627 26754 heartbeater.cc:511] Master 127.22.113.62:36829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:19.160393 26563 ts_manager.cc:194] Registered new tserver with Master: 0c69ff246db5434db50113fc59c536c5 (127.22.113.1:45025)
I20260619 06:53:19.161623 26563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:44085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:19.178956 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:19.295295 26759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:19.295606 26759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:19.295697 26759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:19.299867 26759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:19.299952 26759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:19.300042 26759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:19.305322 26759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:19.306428 26759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:19.307533 26759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:19.317752 26767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.317739 26764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.317752 26765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:19.318202 26759 server_base.cc:1034] running on GCE node
I20260619 06:53:19.318601 26759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:19.319271 26759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:19.320461 26759 hybrid_clock.cc:648] HybridClock initialized: now 1781851999320437 us; error 39 us; skew 500 ppm
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:19.323470 26759 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:19.324366 26759 webserver.cc:460] Webserver started at http://127.22.113.2:37581/ using document root <none> and password file <none>
I20260619 06:53:19.324954 26759 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:19.325017 26759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:19.325250 26759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:19.327353 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "1e1acb75f1c64a05bdac5f7654b09493"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "41666714be3f35bae399f20f907de86f"
server_key_iv: "b6ac92d7dde95427d3dd58efe4c4fe50"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.327915 26759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "1e1acb75f1c64a05bdac5f7654b09493"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "41666714be3f35bae399f20f907de86f"
server_key_iv: "b6ac92d7dde95427d3dd58efe4c4fe50"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.331849 26759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:19.334609 26774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.336102 26759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:19.336242 26759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "1e1acb75f1c64a05bdac5f7654b09493"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "41666714be3f35bae399f20f907de86f"
server_key_iv: "b6ac92d7dde95427d3dd58efe4c4fe50"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.336354 26759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:19.363368 26759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:19.366287 26759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:19.366494 26759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:19.367095 26759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:19.368103 26759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:19.368153 26759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.368227 26759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:19.368273 26759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.378412 26759 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:43985
I20260619 06:53:19.378490 26887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:43985 every 8 connection(s)
I20260619 06:53:19.379434 26759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:19.386616 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26759
I20260619 06:53:19.386770 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:19.387135 22980 external_mini_cluster.cc:1428] Setting key 6b4c4d3e94151f90c9b3d825ba57c245
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:19.395543 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.381664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:43793 (local address 127.22.113.62:36829)
0619 06:53:19.381846 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:19.381858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:19.382823 (+ 965us) server_negotiation.cc:407] Connection header received
0619 06:53:19.384461 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:19.384464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:19.384516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:19.384602 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:19.385923 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.386442 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.388260 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.388629 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.388743 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.391931 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:19.391934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:19.391936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:19.391958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:19.393436 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.393962 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.393973 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.393975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.394044 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.394384 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.394389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.394395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.394560 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:19.394673 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:19.395215 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 06:53:19.395371 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:53:19.396459 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.382114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36829 (local address 127.22.113.2:43793)
0619 06:53:19.382615 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:19.382667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:19.384231 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:19.384774 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:19.384781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:19.385201 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:19.385790 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.385805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.386609 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.386613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.388122 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.388134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.388821 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.388826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.389037 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.389786 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:19.389815 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:19.391734 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:19.393561 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.393567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.393583 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.393851 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.394136 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.394140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.394142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.394280 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.394673 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:19.394680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:19.394969 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 06:53:19.395164 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 06:53:19.395531 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:19.398478 26888 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36829
I20260619 06:53:19.398758 26888 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:19.399304 26888 heartbeater.cc:511] Master 127.22.113.62:36829 requested a full tablet report, sending...
I20260619 06:53:19.400553 26563 ts_manager.cc:194] Registered new tserver with Master: 1e1acb75f1c64a05bdac5f7654b09493 (127.22.113.2:43985)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:53:19.401324 26563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:43793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:19.418812 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:19.544557 26893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:19.544891 26893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:19.544958 26893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:19.549399 26893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:19.549475 26893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:19.549541 26893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:19.555178 26893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:19.556445 26893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:19.557632 26893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:19.568339 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.568377 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:19.568372 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:19.568818 26893 server_base.cc:1034] running on GCE node
I20260619 06:53:19.569316 26893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:19.570043 26893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:19.571229 26893 hybrid_clock.cc:648] HybridClock initialized: now 1781851999571210 us; error 39 us; skew 500 ppm
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:19.574242 26893 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:19.575137 26893 webserver.cc:460] Webserver started at http://127.22.113.3:41205/ using document root <none> and password file <none>
I20260619 06:53:19.575755 26893 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:19.575820 26893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:19.576020 26893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:19.578137 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "249d424f19954abda888fecd69604814"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "5ad2d4bb5a3a21d3ff6efc8a9f02335f"
server_key_iv: "40390c898ed17199c92341b92effffcd"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.578656 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "249d424f19954abda888fecd69604814"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "5ad2d4bb5a3a21d3ff6efc8a9f02335f"
server_key_iv: "40390c898ed17199c92341b92effffcd"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.582716 26893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:19.585465 26908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.587039 26893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:19.587206 26893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "249d424f19954abda888fecd69604814"
format_stamp: "Formatted at 2026-06-19 06:53:19 on dist-test-slave-g8zg"
server_key: "5ad2d4bb5a3a21d3ff6efc8a9f02335f"
server_key_iv: "40390c898ed17199c92341b92effffcd"
server_key_version: "encryptionkey@0"
I20260619 06:53:19.587383 26893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:19.620620 26893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:19.623840 26893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:19.624055 26893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:19.624672 26893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:19.625533 26893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:19.625604 26893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.625680 26893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:19.625725 26893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:19.635949 26893 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:34835
I20260619 06:53:19.635998 27021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:34835 every 8 connection(s)
I20260619 06:53:19.636924 26893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:19.637274 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 26893
I20260619 06:53:19.637434 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:19.637804 22980 external_mini_cluster.cc:1428] Setting key 70f8fe9170100bf9d544d6a0b5281975
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:19.652105 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.638822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:54051 (local address 127.22.113.62:36829)
0619 06:53:19.638961 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:19.638965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:19.640160 (+ 1195us) server_negotiation.cc:407] Connection header received
0619 06:53:19.641779 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:19.641782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:19.641843 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:19.641923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:19.643283 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.643857 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.645177 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.645432 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.645543 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.648412 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:19.648415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:19.648417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:19.648441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:19.649881 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.650367 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.650370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.650372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.650445 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.650828 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.650830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.650832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.651011 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:19.651125 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:19.651793 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 06:53:19.651921 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:53:19.653175 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.639225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36829 (local address 127.22.113.3:54051)
0619 06:53:19.639980 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:19.640051 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:19.641554 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:19.642127 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:19.642136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:19.642544 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:19.643113 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.643125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.643981 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.643985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.645030 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:19.645040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.645586 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.645592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:19.645792 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.646616 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:19.646637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:19.648197 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:19.650000 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.650005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.650017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.650270 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.650530 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:19.650532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:19.650533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:19.650695 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:19.651128 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:19.651138 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:19.651508 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 06:53:19.651753 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 06:53:19.652155 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":200,"threads_started":1}
I20260619 06:53:19.654482 27022 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36829
I20260619 06:53:19.654732 27022 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:19.655359 27022 heartbeater.cc:511] Master 127.22.113.62:36829 requested a full tablet report, sending...
I20260619 06:53:19.656482 26563 ts_manager.cc:194] Registered new tserver with Master: 249d424f19954abda888fecd69604814 (127.22.113.3:34835)
I20260619 06:53:19.657183 26563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:54051
I20260619 06:53:19.664429 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:19.693063 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.682477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.22.113.1:45025)
0619 06:53:19.682885 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:19.682893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:19.682923 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:53:19.682988 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:19.682995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:19.683317 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:19.683479 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:19.684208 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.685038 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.685967 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.686410 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.686507 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.689124 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:19.689178 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:19.689204 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:19.689228 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:19.690978 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.691547 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.691553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.691554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.691632 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.691928 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.691932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.691934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.692256 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:19.692489 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:19.692680 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 06:53:19.692864 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[26518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:19.705878 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:19.696732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.22.113.62:36829)
0619 06:53:19.696933 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:19.696938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:19.697053 (+ 115us) server_negotiation.cc:407] Connection header received
0619 06:53:19.697232 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:19.697236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:19.697293 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:19.697380 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:19.698035 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.698630 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.699485 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:19.699718 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:19.699828 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:19.702513 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:19.702519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:19.702526 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:19.702550 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:19.704161 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.704604 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.704610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.704612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.704691 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:19.704960 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:19.704964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:19.704966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:19.705192 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:19.705350 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:19.705583 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:53:19.705697 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 06:53:19.706276 26563 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090
I20260619 06:53:19.707056 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26625
I20260619 06:53:19.714074 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26759
I20260619 06:53:19.720623 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26893
I20260619 06:53:19.726644 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 26531
2026-06-19T06:53:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2340 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[27034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[27034](info): set up 2 sockets
Jun 19 06:53:19 dist-test-slave-g8zg krb5kdc[27034](info): commencing operation
krb5kdc: starting...
W20260619 06:53:20.791429 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:20 dist-test-slave-g8zg krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:20.922662 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:43513
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:21.045148 27047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:21.045483 27047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:21.045560 27047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:21.049714 27047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:21.049795 27047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:21.049831 27047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:21.049888 27047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:21.049926 27047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:21.055310 27047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:43513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:43513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:21.056422 27047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:21.057507 27047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:21.066608 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.066708 27047 server_base.cc:1034] running on GCE node
W20260619 06:53:21.066602 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.066602 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.067782 27047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:21.069141 27047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:21.070374 27047 hybrid_clock.cc:648] HybridClock initialized: now 1781852001070361 us; error 33 us; skew 500 ppm
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:21.076668 27047 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:21.077675 27047 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:21.077782 27047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:21.078089 27047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:21.080750 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "29195d2eecd844c2ab7290620172a7ec"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "e86c3bd2b90059a6ce26f83a2db9cb0a"
server_key_iv: "dacf569bb191a3c6b9dc4031db7e40a9"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.081430 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "29195d2eecd844c2ab7290620172a7ec"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "e86c3bd2b90059a6ce26f83a2db9cb0a"
server_key_iv: "dacf569bb191a3c6b9dc4031db7e40a9"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.086143 27047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:21.089268 27061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.090883 27047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:21.091041 27047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "29195d2eecd844c2ab7290620172a7ec"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "e86c3bd2b90059a6ce26f83a2db9cb0a"
server_key_iv: "dacf569bb191a3c6b9dc4031db7e40a9"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.091171 27047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:21.102360 27047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:21.105720 27047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:21.105950 27047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:21.114615 27047 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:43513
I20260619 06:53:21.114912 27113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:43513 every 8 connection(s)
I20260619 06:53:21.115497 27047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:21.118517 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:21.120277 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27047
I20260619 06:53:21.120456 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:21.120788 22980 external_mini_cluster.cc:1428] Setting key c24611f8932a738ce40cd2100793e120
I20260619 06:53:21.124887 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: Bootstrap starting.
I20260619 06:53:21.127532 27114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:21.128436 27114 log.cc:824] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:21.130909 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: No bootstrap required, opened a new log
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:21.136497 27114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } }
I20260619 06:53:21.136837 27114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:21.136888 27114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29195d2eecd844c2ab7290620172a7ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:21.137449 27114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } }
I20260619 06:53:21.137614 27114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:21.137686 27114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:21.137796 27114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:21.137981 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.122635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.22.113.62:43513)
0619 06:53:21.123136 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.123154 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.123198 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:53:21.124063 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.124083 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.124519 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.124895 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:21.125868 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.126990 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.128231 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.128833 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.128976 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.131877 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:21.131884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:21.131903 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:21.131937 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:21.134280 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.134789 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.134796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.134803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.134955 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.135235 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.135239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.135267 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.135877 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:21.136109 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.136433 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.136778 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":158,"threads_started":1}
I20260619 06:53:21.139179 27114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } }
I20260619 06:53:21.139643 27114 leader_election.cc:302] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29195d2eecd844c2ab7290620172a7ec; no voters:
I20260619 06:53:21.140038 27114 leader_election.cc:288] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:21.140180 27119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:21.140447 27119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [term 1 LEADER]: Becoming Leader. State: Replica: 29195d2eecd844c2ab7290620172a7ec, State: Running, Role: LEADER
I20260619 06:53:21.140936 27119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } }
I20260619 06:53:21.141522 27114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:21.143333 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29195d2eecd844c2ab7290620172a7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } } }
I20260619 06:53:21.143687 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:21.143560 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29195d2eecd844c2ab7290620172a7ec. Latest consensus state: current_term: 1 leader_uuid: "29195d2eecd844c2ab7290620172a7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29195d2eecd844c2ab7290620172a7ec" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43513 } } }
I20260619 06:53:21.144053 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec [sys.catalog]: This master's current role is: LEADER
W20260619 06:53:21.147287 27134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:21.147378 27134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:21.147496 27133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:21.148227 27133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:21.154362 27133 catalog_manager.cc:1347] Generated new cluster ID: 43460db4c8d543deb46d2c36cc796472
I20260619 06:53:21.154431 27133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:21.180603 27133 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:21.181632 27133 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:21.186400 27133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: Generated new TSK 0
I20260619 06:53:21.187131 27133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:21.224969 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:21.340042 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:21.340338 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:21.340416 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:21.344756 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:21.344836 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:21.344930 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:21.350093 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:21.351161 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:21.352242 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:21.362294 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.362314 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.362294 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.362659 27140 server_base.cc:1034] running on GCE node
I20260619 06:53:21.363337 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:21.364040 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:21.365273 27140 hybrid_clock.cc:648] HybridClock initialized: now 1781852001365233 us; error 55 us; skew 500 ppm
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:21.368281 27140 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:21.369088 27140 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:21.369184 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:21.369427 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:21.371552 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "a9126e0c50534cc89b32088cc34f7ac6"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "0a96d768dc993dcf76ea3c13956fae87"
server_key_iv: "84890bfccc39b5dfe12c3ea24a7ac130"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.372149 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "a9126e0c50534cc89b32088cc34f7ac6"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "0a96d768dc993dcf76ea3c13956fae87"
server_key_iv: "84890bfccc39b5dfe12c3ea24a7ac130"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.376080 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 06:53:21.378621 27154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.380079 27140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:21.380223 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "a9126e0c50534cc89b32088cc34f7ac6"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "0a96d768dc993dcf76ea3c13956fae87"
server_key_iv: "84890bfccc39b5dfe12c3ea24a7ac130"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.380333 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:21.392990 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:21.396270 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:21.396481 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:21.397054 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:21.397974 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:21.398047 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.398125 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:21.398171 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.408874 27140 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:35101
I20260619 06:53:21.408910 27267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:35101 every 8 connection(s)
I20260619 06:53:21.409466 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:21.412653 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27140
I20260619 06:53:21.412815 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:21.413188 22980 external_mini_cluster.cc:1428] Setting key 20bcfd42f6b317e55cc01639bf4584ad
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:21.426056 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.411621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:47989 (local address 127.22.113.62:43513)
0619 06:53:21.411804 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.411808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.413253 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 06:53:21.414257 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.414261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.414320 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.414401 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:21.415737 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.416642 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.418486 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.418925 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.419052 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.422046 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:21.422052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:21.422058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:21.422091 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:21.423926 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.424429 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.424435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.424438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.424531 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.424928 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.424932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.424934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.425207 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:21.425306 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.425780 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.425900 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":73}
I20260619 06:53:21.426967 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.412030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43513 (local address 127.22.113.1:47989)
0619 06:53:21.413054 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.413100 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.414015 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.414609 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.414619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.414998 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:21.415576 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.415591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.416832 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.416839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.418324 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.418334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.419084 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.419087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.419221 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.419953 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:21.419969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:21.421866 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:21.424054 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.424058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.424069 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.424317 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.424631 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.424633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.424634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.424756 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.425320 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:21.425325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:21.425555 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.425713 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.425952 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":180,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:21.428730 27268 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43513
I20260619 06:53:21.429051 27268 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:21.429618 27268 heartbeater.cc:511] Master 127.22.113.62:43513 requested a full tablet report, sending...
I20260619 06:53:21.431445 27078 ts_manager.cc:194] Registered new tserver with Master: a9126e0c50534cc89b32088cc34f7ac6 (127.22.113.1:35101)
I20260619 06:53:21.432602 27078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:47989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:21.446990 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:21.555869 27273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:21.556120 27273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:21.556162 27273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:21.560088 27273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:21.560142 27273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:21.560200 27273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:21.565310 27273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:21.566277 27273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:21.567263 27273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:21.576736 27278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.576792 27279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.576998 27273 server_base.cc:1034] running on GCE node
W20260619 06:53:21.576745 27281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.577633 27273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:21.578457 27273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:21.579666 27273 hybrid_clock.cc:648] HybridClock initialized: now 1781852001579630 us; error 49 us; skew 500 ppm
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:21.582932 27273 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:21.583804 27273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:21.583873 27273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:21.584088 27273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:21.586086 27273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "16d3cb43684b4acebaea1edd66bd016e"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "81384913c9e755c5eea64c67053878cc"
server_key_iv: "91d32635eca1dab923beb93d4014fcc7"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.586719 27273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "16d3cb43684b4acebaea1edd66bd016e"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "81384913c9e755c5eea64c67053878cc"
server_key_iv: "91d32635eca1dab923beb93d4014fcc7"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.590729 27273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:21.593622 27287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.595130 27273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:21.595288 27273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "16d3cb43684b4acebaea1edd66bd016e"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "81384913c9e755c5eea64c67053878cc"
server_key_iv: "91d32635eca1dab923beb93d4014fcc7"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.595413 27273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:21.606504 27273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:21.609668 27273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:21.609871 27273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:21.610455 27273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:21.611294 27273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:21.611368 27273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.611443 27273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:21.611495 27273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.621767 27273 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:34211
I20260619 06:53:21.621829 27400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:34211 every 8 connection(s)
I20260619 06:53:21.622562 27273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:21.624373 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27273
I20260619 06:53:21.624539 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:21.624862 22980 external_mini_cluster.cc:1428] Setting key ab126339e3cd7fefc48c664d2f1252e6
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:21.638623 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.624751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:52663 (local address 127.22.113.62:43513)
0619 06:53:21.624972 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.624981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.626478 (+ 1497us) server_negotiation.cc:407] Connection header received
0619 06:53:21.627845 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.627848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.627900 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.627990 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:21.629587 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.630146 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.631370 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.631796 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.631934 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.634383 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:21.634389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:21.634395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:21.634426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:21.636224 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.636836 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.636839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.636841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.636933 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.637340 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.637343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.637345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.637563 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:21.637666 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.638262 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.638395 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:53:21.639547 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.625263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43513 (local address 127.22.113.2:52663)
0619 06:53:21.626251 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.626294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.627532 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.628232 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.628244 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.628813 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:21.629399 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.629422 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.630273 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.630276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.631208 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.631222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.631976 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.631979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.632118 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.632745 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:21.632762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:21.634196 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:21.636341 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.636346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.636357 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.636634 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.637067 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.637072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.637076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.637224 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.637667 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:21.637694 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:21.637940 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.638158 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.638403 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":172,"threads_started":1}
I20260619 06:53:21.640846 27401 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43513
I20260619 06:53:21.641287 27401 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:21.641896 27401 heartbeater.cc:511] Master 127.22.113.62:43513 requested a full tablet report, sending...
I20260619 06:53:21.643206 27078 ts_manager.cc:194] Registered new tserver with Master: 16d3cb43684b4acebaea1edd66bd016e (127.22.113.2:34211)
I20260619 06:53:21.643923 27078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:52663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:21.658718 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:21.788525 27406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:21.788869 27406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:21.788954 27406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:21.793267 27406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:21.793382 27406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:21.793488 27406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:21.799455 27406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:21.800833 27406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:21.802074 27406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:21.812922 27414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.812953 27412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:21.812922 27411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:21.813194 27406 server_base.cc:1034] running on GCE node
I20260619 06:53:21.813859 27406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:21.814630 27406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:21.815831 27406 hybrid_clock.cc:648] HybridClock initialized: now 1781852001815792 us; error 52 us; skew 500 ppm
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:21.818753 27406 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:21.819561 27406 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:21.819638 27406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:21.819832 27406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:21.821887 27406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "66975cb2bea84c6ca2c1e201cd8dc6db"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "35396db765bf0dfc482991fa3fde963d"
server_key_iv: "c667e1f01836f823e34f0936e70485de"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.822518 27406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "66975cb2bea84c6ca2c1e201cd8dc6db"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "35396db765bf0dfc482991fa3fde963d"
server_key_iv: "c667e1f01836f823e34f0936e70485de"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.826390 27406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:21.828934 27420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.830431 27406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 06:53:21.830544 27406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "66975cb2bea84c6ca2c1e201cd8dc6db"
format_stamp: "Formatted at 2026-06-19 06:53:21 on dist-test-slave-g8zg"
server_key: "35396db765bf0dfc482991fa3fde963d"
server_key_iv: "c667e1f01836f823e34f0936e70485de"
server_key_version: "encryptionkey@0"
I20260619 06:53:21.830636 27406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:21.840911 27406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:21.844367 27406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:21.844527 27406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:21.845050 27406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:21.845976 27406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:21.846047 27406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.846115 27406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:21.846158 27406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:21.856292 27406 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:33799
I20260619 06:53:21.856338 27533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:33799 every 8 connection(s)
I20260619 06:53:21.856892 27406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:21.865970 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27406
I20260619 06:53:21.866142 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:21.866495 22980 external_mini_cluster.cc:1428] Setting key 1f13479d4f9527d66203bbd015f4bc17
Jun 19 06:53:21 dist-test-slave-g8zg krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:21.871889 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.858913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:36805 (local address 127.22.113.62:43513)
0619 06:53:21.859112 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.859118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.860345 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 06:53:21.861321 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.861324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.861379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.861478 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:21.862837 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.863407 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.864672 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.864934 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.865042 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.868059 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:21.868063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:21.868065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:21.868088 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:21.869764 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.870312 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.870317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.870318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.870402 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.870741 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.870746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.870748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.870931 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:21.871038 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.871587 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.871715 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:53:21.872666 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.859281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43513 (local address 127.22.113.3:36805)
0619 06:53:21.860156 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.860205 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.861104 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.861670 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.861677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.862092 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:21.862677 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.862691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.863530 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.863533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.864520 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.864529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.865096 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.865099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.865250 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.866064 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:21.866085 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:21.867848 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:21.869929 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.869940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.869951 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.870196 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.870521 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:21.870524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:21.870525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:21.870647 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:21.871036 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:21.871042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:21.871354 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.871536 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.871860 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260619 06:53:21.874197 27534 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43513
I20260619 06:53:21.874477 27534 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:21.874871 27534 heartbeater.cc:511] Master 127.22.113.62:43513 requested a full tablet report, sending...
I20260619 06:53:21.875934 27078 ts_manager.cc:194] Registered new tserver with Master: 66975cb2bea84c6ca2c1e201cd8dc6db (127.22.113.3:33799)
I20260619 06:53:21.876457 27078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:36805
I20260619 06:53:21.880945 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:53:21.890808 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.883339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.22.113.62:43513)
0619 06:53:21.883485 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.883489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.883680 (+ 191us) server_negotiation.cc:407] Connection header received
0619 06:53:21.883836 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.883838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.883898 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.884009 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:21.884738 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.885289 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.886127 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.886326 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.886402 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.887883 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:21.887886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:21.887888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:21.887909 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:21.889372 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.889773 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.889776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.889777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.889849 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:21.890090 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:21.890092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:21.890093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:21.890287 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:21.890390 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.890603 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.890692 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:53:21.893939 27078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:21.897244 27078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:21.913086 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.907246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33799 (local address 127.0.0.1:59130)
0619 06:53:21.907537 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.907552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.907637 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.908431 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.908435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.908455 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:21.908585 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.908593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.909952 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.909955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.911342 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.911355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912517 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.912521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.912652 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.912679 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.912743 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.912938 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":101,"thread_start_us":128,"threads_started":1}
I20260619 06:53:21.913069 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.907804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.22.113.1:35101)
0619 06:53:21.908178 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.908185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.908486 (+ 301us) server_negotiation.cc:407] Connection header received
0619 06:53:21.908703 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.908709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.908887 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.909014 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:21.909481 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.910402 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.911795 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912340 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.912466 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.912633 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.912802 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.912906 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260619 06:53:21.913285 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.906433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:34211 (local address 127.0.0.1:39404)
0619 06:53:21.906801 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.906846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.906968 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.907778 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.907785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.907815 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:21.908045 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.908054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.909712 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.909715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.911356 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.911363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912517 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.912521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.912904 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.912920 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.913009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.913095 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":382,"thread_start_us":135,"threads_started":1}
I20260619 06:53:21.913645 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.908118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:35101 (local address 127.0.0.1:56676)
0619 06:53:21.908376 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.908392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.908555 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.909189 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.909192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.909209 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:21.909316 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.909321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.910543 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.910548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.911663 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.911675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912465 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.912469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.912588 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.912627 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.912762 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.913507 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260619 06:53:21.913693 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.906675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.22.113.2:34211)
0619 06:53:21.907075 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.907089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.907119 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:53:21.907218 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.907223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.907531 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.908222 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:21.908395 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.909576 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.911492 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912369 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.912647 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.912822 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.913045 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.913163 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":181,"threads_started":1}
I20260619 06:53:21.913915 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.907491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.22.113.3:33799)
0619 06:53:21.907928 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.907941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.907966 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:53:21.908040 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.908047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.908275 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.908739 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:21.908884 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.909809 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.911503 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.912363 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.913439 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.913579 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.913628 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.913734 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":200,"threads_started":1}
I20260619 06:53:21.916915 27468 tablet_service.cc:1467] Processing CreateTablet for tablet f204c6690f864c07b35360834fad8b4f (DEFAULT_TABLE table=test-table [id=f8f3d234608242f6915aa9daf735e1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:21.916913 27202 tablet_service.cc:1467] Processing CreateTablet for tablet f204c6690f864c07b35360834fad8b4f (DEFAULT_TABLE table=test-table [id=f8f3d234608242f6915aa9daf735e1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:21.917008 27335 tablet_service.cc:1467] Processing CreateTablet for tablet f204c6690f864c07b35360834fad8b4f (DEFAULT_TABLE table=test-table [id=f8f3d234608242f6915aa9daf735e1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:21.918083 27202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f204c6690f864c07b35360834fad8b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:21.918090 27468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f204c6690f864c07b35360834fad8b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:21.918105 27335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f204c6690f864c07b35360834fad8b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:21.925380 27551 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Bootstrap starting.
I20260619 06:53:21.926436 27553 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: Bootstrap starting.
I20260619 06:53:21.926436 27552 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: Bootstrap starting.
I20260619 06:53:21.928325 27551 tablet_bootstrap.cc:654] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:21.928807 27552 tablet_bootstrap.cc:654] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:21.928807 27553 tablet_bootstrap.cc:654] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:21.929587 27551 log.cc:824] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:21.929759 27553 log.cc:824] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:21.929889 27552 log.cc:824] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:21.931952 27553 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: No bootstrap required, opened a new log
I20260619 06:53:21.932158 27553 ts_tablet_manager.cc:1397] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 06:53:21.932233 27552 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: No bootstrap required, opened a new log
I20260619 06:53:21.932462 27552 ts_tablet_manager.cc:1397] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:53:21.932914 27551 tablet_bootstrap.cc:492] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: No bootstrap required, opened a new log
I20260619 06:53:21.933091 27551 ts_tablet_manager.cc:1397] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 06:53:21.936012 27553 raft_consensus.cc:348] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.936326 27553 raft_consensus.cc:374] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:21.936381 27553 raft_consensus.cc:729] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9126e0c50534cc89b32088cc34f7ac6, State: Initialized, Role: FOLLOWER
I20260619 06:53:21.936854 27552 raft_consensus.cc:348] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.937158 27552 raft_consensus.cc:374] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:21.937213 27552 raft_consensus.cc:729] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66975cb2bea84c6ca2c1e201cd8dc6db, State: Initialized, Role: FOLLOWER
I20260619 06:53:21.937242 27553 consensus_queue.cc:260] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.937649 27552 consensus_queue.cc:260] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.938306 27553 ts_tablet_manager.cc:1428] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:53:21.938688 27552 ts_tablet_manager.cc:1428] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 06:53:21.938642 27551 raft_consensus.cc:348] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.938827 27268 heartbeater.cc:503] Master 127.22.113.62:43513 was elected leader, sending a full tablet report...
I20260619 06:53:21.938894 27551 raft_consensus.cc:374] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:21.938946 27551 raft_consensus.cc:729] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d3cb43684b4acebaea1edd66bd016e, State: Initialized, Role: FOLLOWER
I20260619 06:53:21.939043 27534 heartbeater.cc:503] Master 127.22.113.62:43513 was elected leader, sending a full tablet report...
I20260619 06:53:21.939380 27551 consensus_queue.cc:260] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.940263 27401 heartbeater.cc:503] Master 127.22.113.62:43513 was elected leader, sending a full tablet report...
I20260619 06:53:21.942190 27551 ts_tablet_manager.cc:1428] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:53:21.970353 27558 raft_consensus.cc:482] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:21.970578 27558 raft_consensus.cc:504] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.971746 27558 leader_election.cc:288] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d3cb43684b4acebaea1edd66bd016e (127.22.113.2:34211), a9126e0c50534cc89b32088cc34f7ac6 (127.22.113.1:35101)
I20260619 06:53:21.976470 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.972131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:34211 (local address 127.22.113.3:34745)
0619 06:53:21.972304 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.972320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.972462 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.972722 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.972725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.972740 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:21.972865 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.972873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.973978 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.973982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.975020 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.975029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.975994 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.975998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.976117 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.976148 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.976236 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.976315 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 06:53:21.976490 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.972283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:34745 (local address 127.22.113.2:34211)
0619 06:53:21.972452 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.972457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.972475 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:21.972562 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.972567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.972617 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.972710 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:21.973033 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.973854 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.975158 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.975832 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.975984 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.976037 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.976290 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.976360 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:53:21.977246 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.972510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:35101 (local address 127.22.113.3:38735)
0619 06:53:21.973046 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:21.973062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:21.973213 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:21.973479 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:21.973483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:21.973499 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:21.973629 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.973634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.974621 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.974625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.975828 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:21.975838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.976810 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.976814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:21.976896 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.976913 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:53:21.977021 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:53:21.977112 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":173,"threads_started":1}
I20260619 06:53:21.977267 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:21.972682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:38735 (local address 127.22.113.1:35101)
0619 06:53:21.972952 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:21.972957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:21.973184 (+ 227us) server_negotiation.cc:407] Connection header received
0619 06:53:21.973328 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:21.973330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:21.973370 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:21.973461 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:21.973740 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.974489 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.975970 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:21.976544 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:21.976670 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:21.976730 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:21.977066 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:53:21.977158 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260619 06:53:21.977299 27355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f204c6690f864c07b35360834fad8b4f" candidate_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d3cb43684b4acebaea1edd66bd016e" is_pre_election: true
I20260619 06:53:21.977622 27355 raft_consensus.cc:2454] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66975cb2bea84c6ca2c1e201cd8dc6db in term 0.
I20260619 06:53:21.978148 27222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f204c6690f864c07b35360834fad8b4f" candidate_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9126e0c50534cc89b32088cc34f7ac6" is_pre_election: true
I20260619 06:53:21.978212 27422 leader_election.cc:302] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d3cb43684b4acebaea1edd66bd016e, 66975cb2bea84c6ca2c1e201cd8dc6db; no voters:
I20260619 06:53:21.978405 27222 raft_consensus.cc:2454] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66975cb2bea84c6ca2c1e201cd8dc6db in term 0.
I20260619 06:53:21.978566 27558 raft_consensus.cc:2781] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:21.978681 27558 raft_consensus.cc:482] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:21.978754 27558 raft_consensus.cc:3037] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:21.980104 27558 raft_consensus.cc:504] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.980477 27558 leader_election.cc:288] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [CANDIDATE]: Term 1 election: Requested vote from peers 16d3cb43684b4acebaea1edd66bd016e (127.22.113.2:34211), a9126e0c50534cc89b32088cc34f7ac6 (127.22.113.1:35101)
I20260619 06:53:21.980964 27222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f204c6690f864c07b35360834fad8b4f" candidate_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9126e0c50534cc89b32088cc34f7ac6"
I20260619 06:53:21.980964 27355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f204c6690f864c07b35360834fad8b4f" candidate_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d3cb43684b4acebaea1edd66bd016e"
I20260619 06:53:21.981107 27355 raft_consensus.cc:3037] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:21.981107 27222 raft_consensus.cc:3037] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:21.982546 27222 raft_consensus.cc:2454] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66975cb2bea84c6ca2c1e201cd8dc6db in term 1.
I20260619 06:53:21.982546 27355 raft_consensus.cc:2454] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66975cb2bea84c6ca2c1e201cd8dc6db in term 1.
I20260619 06:53:21.982869 27422 leader_election.cc:302] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d3cb43684b4acebaea1edd66bd016e, 66975cb2bea84c6ca2c1e201cd8dc6db; no voters:
I20260619 06:53:21.983095 27558 raft_consensus.cc:2781] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:21.983371 27558 raft_consensus.cc:686] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 1 LEADER]: Becoming Leader. State: Replica: 66975cb2bea84c6ca2c1e201cd8dc6db, State: Running, Role: LEADER
I20260619 06:53:21.983717 27558 consensus_queue.cc:237] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } }
I20260619 06:53:21.987519 27077 catalog_manager.cc:5495] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db reported cstate change: term changed from 0 to 1, leader changed from <none> to 66975cb2bea84c6ca2c1e201cd8dc6db (127.22.113.3). New cstate: current_term: 1 leader_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66975cb2bea84c6ca2c1e201cd8dc6db" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:22.015579 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.011644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.22.113.3:33799)
0619 06:53:22.011781 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.011785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.011825 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:53:22.011998 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.012000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.012056 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.012178 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:22.012535 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:22.013052 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:22.014005 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:22.014240 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:22.014329 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:22.014672 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:22.015105 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:22.015201 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.015416 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.015469 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:53:22.022333 27355 raft_consensus.cc:1261] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 1 FOLLOWER]: Refusing update from remote peer 66975cb2bea84c6ca2c1e201cd8dc6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:22.022333 27222 raft_consensus.cc:1261] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6 [term 1 FOLLOWER]: Refusing update from remote peer 66975cb2bea84c6ca2c1e201cd8dc6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:22.023054 27558 consensus_queue.cc:1035] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9126e0c50534cc89b32088cc34f7ac6" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:22.023288 27561 consensus_queue.cc:1035] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d3cb43684b4acebaea1edd66bd016e" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:22.030958 27564 mvcc.cc:205] Tried to move back new op lower bound from 7298465800275779584 to 7298465800128475136. Current Snapshot: MvccSnapshot[applied={T|T < 7298465800275779584}]
I20260619 06:53:22.032162 27565 mvcc.cc:205] Tried to move back new op lower bound from 7298465800275779584 to 7298465800128475136. Current Snapshot: MvccSnapshot[applied={T|T < 7298465800275779584}]
I20260619 06:53:22.042135 27077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:22.042400 27077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:22.045342 27077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29195d2eecd844c2ab7290620172a7ec: Sending DeleteTablet for 3 replicas of tablet f204c6690f864c07b35360834fad8b4f
I20260619 06:53:22.046262 27335 tablet_service.cc:1514] Processing DeleteTablet for tablet f204c6690f864c07b35360834fad8b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:22 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:39404
I20260619 06:53:22.046339 27468 tablet_service.cc:1514] Processing DeleteTablet for tablet f204c6690f864c07b35360834fad8b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:22 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59130
I20260619 06:53:22.046522 27202 tablet_service.cc:1514] Processing DeleteTablet for tablet f204c6690f864c07b35360834fad8b4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:22 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:56676
I20260619 06:53:22.046953 27573 tablet_replica.cc:330] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: stopping tablet replica
I20260619 06:53:22.047358 27573 raft_consensus.cc:2229] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:22.047577 27575 tablet_replica.cc:330] T f204c6690f864c07b35360834fad8b4f P a9126e0c50534cc89b32088cc34f7ac6: stopping tablet replica
I20260619 06:53:22.047639 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27140
I20260619 06:53:22.049273 27573 raft_consensus.cc:2258] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:22.051059 27573 ts_tablet_manager.cc:1905] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:22.053846 27573 ts_tablet_manager.cc:1918] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:22.053957 27573 log.cc:1196] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/f204c6690f864c07b35360834fad8b4f
I20260619 06:53:22.054323 27573 ts_tablet_manager.cc:1939] T f204c6690f864c07b35360834fad8b4f P 16d3cb43684b4acebaea1edd66bd016e: Deleting consensus metadata
I20260619 06:53:22.055744 27063 catalog_manager.cc:4841] TS 16d3cb43684b4acebaea1edd66bd016e (127.22.113.2:34211): tablet f204c6690f864c07b35360834fad8b4f (table test-table [id=f8f3d234608242f6915aa9daf735e1e9]) successfully deleted
W20260619 06:53:22.056052 27063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:35101 (error 108)
I20260619 06:53:22.056100 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27273
I20260619 06:53:22.056939 27574 tablet_replica.cc:330] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db: stopping tablet replica
I20260619 06:53:22.057291 27574 raft_consensus.cc:2229] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:22.057742 27574 raft_consensus.cc:2258] T f204c6690f864c07b35360834fad8b4f P 66975cb2bea84c6ca2c1e201cd8dc6db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:22.058894 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.058426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:35101 (local address 127.0.0.1:56692)
0619 06:53:22.058666 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.058768 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 06:53:22.059089 27063 catalog_manager.cc:4584] TS a9126e0c50534cc89b32088cc34f7ac6 (127.22.113.1:35101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f204c6690f864c07b35360834fad8b4f: Network error: Client connection negotiation failed: client connection to 127.22.113.1:35101: connect: Connection refused (error 111)
I20260619 06:53:22.063588 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27406
I20260619 06:53:22.070456 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27047
2026-06-19T06:53:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2344 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:22Z Disabled control of system clock
I20260619 06:53:22.104622 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40681
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:53:22.229209 27580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:22.229532 27580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:22.229575 27580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:22.233927 27580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:22.234006 27580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:22.234022 27580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:22.234040 27580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:22.234056 27580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:22.239497 27580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:22.240993 27580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:22.242751 27580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:22.253110 27585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.253110 27588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.253196 27580 server_base.cc:1034] running on GCE node
W20260619 06:53:22.253114 27586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.254110 27580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:22.255424 27580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:22.256637 27580 hybrid_clock.cc:648] HybridClock initialized: now 1781852002256611 us; error 49 us; skew 500 ppm
I20260619 06:53:22.258713 27580 webserver.cc:460] Webserver started at http://127.22.113.62:44505/ using document root <none> and password file <none>
I20260619 06:53:22.259407 27580 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:22.259507 27580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:22.259727 27580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:22.261894 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "8ffda89361444f1e903d33200730718b"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "2b38c10455becc9820dda4cec80f6f0f"
server_key_iv: "b2116dc7b4c2325de5801f0593d66b5a"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.262490 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "8ffda89361444f1e903d33200730718b"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "2b38c10455becc9820dda4cec80f6f0f"
server_key_iv: "b2116dc7b4c2325de5801f0593d66b5a"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.266391 27580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:22.269160 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.270556 27580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:22.270722 27580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "8ffda89361444f1e903d33200730718b"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "2b38c10455becc9820dda4cec80f6f0f"
server_key_iv: "b2116dc7b4c2325de5801f0593d66b5a"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.270845 27580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:22.275561 27580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:22.276203 27580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:22.276392 27580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:22.284015 27580 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40681
I20260619 06:53:22.284026 27646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40681 every 8 connection(s)
I20260619 06:53:22.284986 27580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:22.288851 27647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:22.291316 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27580
I20260619 06:53:22.291466 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:22.291828 22980 external_mini_cluster.cc:1428] Setting key 0112eb2e7f94e6b20af78ee4e2254525
I20260619 06:53:22.296301 27647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: Bootstrap starting.
I20260619 06:53:22.298652 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.293713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.22.113.62:40681)
0619 06:53:22.294628 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.294648 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.294689 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:53:22.295629 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.295651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.296060 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.296419 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.297296 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.297301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.297313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.297323 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.297385 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.297402 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.297573 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.297645 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.297887 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":192,"threads_started":1}
I20260619 06:53:22.299772 27647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:22.300709 27647 log.cc:824] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:22.303344 27647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: No bootstrap required, opened a new log
I20260619 06:53:22.307442 27647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } }
I20260619 06:53:22.307713 27647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:22.307773 27647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffda89361444f1e903d33200730718b, State: Initialized, Role: FOLLOWER
I20260619 06:53:22.308265 27647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } }
I20260619 06:53:22.308404 27647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:22.308497 27647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:22.308620 27647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:22.309763 27647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } }
I20260619 06:53:22.310156 27647 leader_election.cc:302] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ffda89361444f1e903d33200730718b; no voters:
I20260619 06:53:22.310534 27647 leader_election.cc:288] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:22.310722 27652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:22.311038 27652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [term 1 LEADER]: Becoming Leader. State: Replica: 8ffda89361444f1e903d33200730718b, State: Running, Role: LEADER
I20260619 06:53:22.311491 27652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } }
I20260619 06:53:22.312064 27647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:22.313411 27654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ffda89361444f1e903d33200730718b. Latest consensus state: current_term: 1 leader_uuid: "8ffda89361444f1e903d33200730718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } } }
I20260619 06:53:22.313568 27654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:22.313890 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ffda89361444f1e903d33200730718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffda89361444f1e903d33200730718b" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40681 } } }
I20260619 06:53:22.314013 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:22.313974 27661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:22.317603 27661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:22.324169 27661 catalog_manager.cc:1347] Generated new cluster ID: b9144a4d1fd842bebae2857c571e9dc5
I20260619 06:53:22.324254 27661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:22.336114 27661 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:22.336472 27661 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:22.343420 27661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: Generated new TSK 0
I20260619 06:53:22.344398 27661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:22.355122 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--builtin_ntp_servers=127.22.113.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:53:22.494554 27671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:22.494910 27671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:22.494993 27671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:22.499215 27671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:22.499342 27671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:22.499439 27671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:22.504475 27671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:22.505808 27671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:22.507499 27671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:22.519701 27679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.519706 27676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.519701 27677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.520114 27671 server_base.cc:1034] running on GCE node
I20260619 06:53:22.520805 27671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:22.521588 27671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:22.522799 27671 hybrid_clock.cc:648] HybridClock initialized: now 1781852002522757 us; error 59 us; skew 500 ppm
I20260619 06:53:22.525229 27671 webserver.cc:460] Webserver started at http://127.22.113.1:37215/ using document root <none> and password file <none>
I20260619 06:53:22.525995 27671 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:22.526067 27671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:22.526335 27671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:22.528893 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "19db378a96844d5fb39b974c531b7570"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a7531cb0a96f77732c4f4eeadf849a9a"
server_key_iv: "e575695392cb2e6d57b1383b861bb280"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.529539 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "19db378a96844d5fb39b974c531b7570"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a7531cb0a96f77732c4f4eeadf849a9a"
server_key_iv: "e575695392cb2e6d57b1383b861bb280"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.534034 27671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:22.537155 27685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.538714 27671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:22.538861 27671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "19db378a96844d5fb39b974c531b7570"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a7531cb0a96f77732c4f4eeadf849a9a"
server_key_iv: "e575695392cb2e6d57b1383b861bb280"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.538995 27671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:22.544250 27671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:22.544955 27671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:22.545159 27671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:22.545786 27671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:22.546824 27671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:22.546875 27671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.546916 27671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:22.546974 27671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.558393 27671 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46529
I20260619 06:53:22.558440 27798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46529 every 8 connection(s)
I20260619 06:53:22.559506 27671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:22.563673 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27671
I20260619 06:53:22.563774 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:22.564082 22980 external_mini_cluster.cc:1428] Setting key 8d79369a83455d59066564c0f5aeb0b0
I20260619 06:53:22.565954 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--builtin_ntp_servers=127.22.113.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:53:22.566565 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.561612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:50121 (local address 127.22.113.62:40681)
0619 06:53:22.561812 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.561817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.562949 (+ 1132us) server_negotiation.cc:407] Connection header received
0619 06:53:22.564578 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.564581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.564632 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.564746 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.566010 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.566013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.566016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.566028 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.566052 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.566063 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.566178 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.566361 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.566413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:53:22.567781 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.561992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40681 (local address 127.22.113.1:50121)
0619 06:53:22.562767 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.562814 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.564347 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.564929 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.564937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.565065 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.565733 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.565735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.565784 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.565855 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.566217 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.566226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.566232 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.566412 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.566673 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260619 06:53:22.569144 27799 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40681
I20260619 06:53:22.569428 27799 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:22.569976 27799 heartbeater.cc:511] Master 127.22.113.62:40681 requested a full tablet report, sending...
I20260619 06:53:22.571954 27611 ts_manager.cc:194] Registered new tserver with Master: 19db378a96844d5fb39b974c531b7570 (127.22.113.1:46529)
W20260619 06:53:22.683149 27802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:22.683439 27802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:22.683477 27802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:22.687520 27802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:22.687573 27802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:22.687630 27802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:22.692591 27802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:22.693718 27802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:22.695130 27802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:22.705340 27807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.705359 27808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.705363 27810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.705597 27802 server_base.cc:1034] running on GCE node
I20260619 06:53:22.706180 27802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:22.706915 27802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:22.708107 27802 hybrid_clock.cc:648] HybridClock initialized: now 1781852002708083 us; error 50 us; skew 500 ppm
I20260619 06:53:22.710228 27802 webserver.cc:460] Webserver started at http://127.22.113.2:40029/ using document root <none> and password file <none>
I20260619 06:53:22.710872 27802 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:22.710937 27802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:22.711158 27802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:22.713553 27802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "f429460a04b64d3dbab4fabf9c6d4914"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "042138b22329805202ab3e5e58949aad"
server_key_iv: "dfb5617895651ec5d0b2224f8f98ad68"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.714243 27802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "f429460a04b64d3dbab4fabf9c6d4914"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "042138b22329805202ab3e5e58949aad"
server_key_iv: "dfb5617895651ec5d0b2224f8f98ad68"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.718225 27802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:53:22.721093 27816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.722506 27802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:22.722690 27802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "f429460a04b64d3dbab4fabf9c6d4914"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "042138b22329805202ab3e5e58949aad"
server_key_iv: "dfb5617895651ec5d0b2224f8f98ad68"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.722837 27802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:22.727495 27802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:22.728078 27802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:22.728219 27802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:22.728709 27802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:22.729573 27802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:22.729630 27802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.729668 27802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:22.729684 27802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.739708 27802 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:39665
I20260619 06:53:22.739756 27929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:39665 every 8 connection(s)
I20260619 06:53:22.740696 27802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:22.743459 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27802
I20260619 06:53:22.743676 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:22.744100 22980 external_mini_cluster.cc:1428] Setting key 2e0b12980903aa782881147472beb087
I20260619 06:53:22.746654 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--builtin_ntp_servers=127.22.113.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:53:22.748818 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.743133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:40829 (local address 127.22.113.62:40681)
0619 06:53:22.743410 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.743416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.744738 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 06:53:22.746602 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.746605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.746649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.746730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.748204 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.748207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.748209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.748219 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.748239 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.748247 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.748326 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.748576 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.748636 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 06:53:22.749917 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.743533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40681 (local address 127.22.113.2:40829)
0619 06:53:22.744521 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.744595 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.746337 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.746951 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.746964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.747092 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.747843 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.747847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.747923 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.748036 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.748346 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.748357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.748366 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.748596 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.748876 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":135,"threads_started":1}
I20260619 06:53:22.751479 27930 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40681
I20260619 06:53:22.751770 27930 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:22.752274 27930 heartbeater.cc:511] Master 127.22.113.62:40681 requested a full tablet report, sending...
I20260619 06:53:22.753536 27611 ts_manager.cc:194] Registered new tserver with Master: f429460a04b64d3dbab4fabf9c6d4914 (127.22.113.2:39665)
W20260619 06:53:22.857658 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:22.857924 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:22.857964 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:22.861965 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:22.862027 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:22.862082 27933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:22.866928 27933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:22.868008 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:22.869367 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:22.879428 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:22.879431 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.879624 27933 server_base.cc:1034] running on GCE node
W20260619 06:53:22.879480 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:22.880342 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:22.881050 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:22.882243 27933 hybrid_clock.cc:648] HybridClock initialized: now 1781852002882226 us; error 35 us; skew 500 ppm
I20260619 06:53:22.884204 27933 webserver.cc:460] Webserver started at http://127.22.113.3:38649/ using document root <none> and password file <none>
I20260619 06:53:22.884791 27933 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:22.884888 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:22.885123 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:22.887143 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a2840e7aa447570171527e98506dbf6e"
server_key_iv: "ba2229817ef91268402ed3a0e91442db"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.887753 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a2840e7aa447570171527e98506dbf6e"
server_key_iv: "ba2229817ef91268402ed3a0e91442db"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.891490 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:22.894228 27947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.895761 27933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:22.895915 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd"
format_stamp: "Formatted at 2026-06-19 06:53:22 on dist-test-slave-g8zg"
server_key: "a2840e7aa447570171527e98506dbf6e"
server_key_iv: "ba2229817ef91268402ed3a0e91442db"
server_key_version: "encryptionkey@0"
I20260619 06:53:22.896041 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:22.901119 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:22.901760 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:22.901973 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:22.902544 27933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:22.903599 27933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:22.903694 27933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.903772 27933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:22.903827 27933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:22.914710 27933 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:37951
I20260619 06:53:22.914758 28060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:37951 every 8 connection(s)
I20260619 06:53:22.915750 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:22.922602 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.917947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:56785 (local address 127.22.113.62:40681)
0619 06:53:22.918121 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.918126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.919289 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 06:53:22.920772 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.920775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.920825 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.920912 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.922078 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.922083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.922087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.922099 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.922122 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.922131 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.922251 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.922394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.922442 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:53:22.923650 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.918339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40681 (local address 127.22.113.3:56785)
0619 06:53:22.919091 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.919137 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.920546 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.921080 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.921088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.921182 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.921812 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.921813 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.921856 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.921924 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.922229 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.922238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.922244 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.922444 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.922697 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260619 06:53:22.924312 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 27933
I20260619 06:53:22.924453 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:22.924842 22980 external_mini_cluster.cc:1428] Setting key 88ae24508e6d7d2b5b7854b27a479544
I20260619 06:53:22.925151 28061 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40681
I20260619 06:53:22.925518 28061 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:22.926084 28061 heartbeater.cc:511] Master 127.22.113.62:40681 requested a full tablet report, sending...
I20260619 06:53:22.927386 27611 ts_manager.cc:194] Registered new tserver with Master: 9a374d7b1b3a4942a4f51f4a70ab73bd (127.22.113.3:37951)
I20260619 06:53:22.939320 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:53:22.943879 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.941661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.22.113.62:40681)
0619 06:53:22.941839 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.941845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.942082 (+ 237us) server_negotiation.cc:407] Connection header received
0619 06:53:22.942290 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.942293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.942345 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.942428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.943390 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.943397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.943399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.943412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.943437 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.943452 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.943535 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.943627 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.943666 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:22.947155 27611 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:22.950796 27611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:22.964311 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.960215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:39665 (local address 127.0.0.1:59412)
0619 06:53:22.960680 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.960737 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.960842 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.962523 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.962531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.962544 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.963017 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.963021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.963035 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.963060 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.963997 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.964001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.964005 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.964100 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.964145 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":130,"threads_started":1}
I20260619 06:53:22.964570 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.961112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.22.113.3:37951)
0619 06:53:22.961657 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.961675 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.961702 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:53:22.961759 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.961763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.961963 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.962110 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.963309 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.963315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.963328 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.963343 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.963393 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.963418 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.963565 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.964147 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.964294 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":167,"threads_started":1}
I20260619 06:53:22.964993 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.960428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.22.113.2:39665)
0619 06:53:22.962035 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.962066 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.962090 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:53:22.962145 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.962150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.962350 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.962743 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.963726 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.963732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.963744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.963753 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.963810 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.963838 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.963995 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.964734 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.964831 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":131,"threads_started":1}
I20260619 06:53:22.965155 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.960560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37951 (local address 127.0.0.1:58136)
0619 06:53:22.961423 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.961439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.961532 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.962142 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.962148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.962236 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.962868 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.962871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.962893 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.962972 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.963543 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.963553 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.963560 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.963688 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.963762 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":166,"threads_started":1}
I20260619 06:53:22.966094 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.961052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46529 (local address 127.0.0.1:41924)
0619 06:53:22.961721 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:22.961733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:22.961811 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:22.962862 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:22.962868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:22.962893 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:22.963507 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:22.963511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:22.963527 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:22.963557 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:22.965821 (+ 2264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:22.965823 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:22.965826 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:53:22.965926 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:53:22.965972 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":100,"threads_started":1}
I20260619 06:53:22.966180 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:22.961226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.22.113.1:46529)
0619 06:53:22.961779 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:22.961794 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:22.961822 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:53:22.961961 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:22.961968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:22.962118 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:22.962241 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:22.963684 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:22.963689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:22.963699 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:22.963710 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:22.963753 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:22.963771 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:22.963881 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:22.965955 (+ 2074us) server_negotiation.cc:299] Negotiation successful
0619 06:53:22.966022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":96,"threads_started":1}
I20260619 06:53:22.968559 27864 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c7bbad44d48deb92f904c88b6197a (DEFAULT_TABLE table=test-table [id=f0a6062765bd4d27ac71a2ce65c8914c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:22.968616 27733 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c7bbad44d48deb92f904c88b6197a (DEFAULT_TABLE table=test-table [id=f0a6062765bd4d27ac71a2ce65c8914c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:22.969849 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c7bbad44d48deb92f904c88b6197a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:22.969849 27864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c7bbad44d48deb92f904c88b6197a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:22.970676 27995 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c7bbad44d48deb92f904c88b6197a (DEFAULT_TABLE table=test-table [id=f0a6062765bd4d27ac71a2ce65c8914c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:22.971728 27995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c7bbad44d48deb92f904c88b6197a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:22.978363 28078 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Bootstrap starting.
I20260619 06:53:22.978874 28079 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Bootstrap starting.
I20260619 06:53:22.979354 28080 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Bootstrap starting.
I20260619 06:53:22.981686 28078 tablet_bootstrap.cc:654] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:22.981686 28080 tablet_bootstrap.cc:654] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:22.982615 28080 log.cc:824] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:22.982615 28078 log.cc:824] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:22.984045 28079 tablet_bootstrap.cc:654] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:22.984858 28078 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: No bootstrap required, opened a new log
I20260619 06:53:22.984858 28080 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: No bootstrap required, opened a new log
I20260619 06:53:22.984901 28079 log.cc:824] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:22.985066 28080 ts_tablet_manager.cc:1397] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:53:22.985066 28078 ts_tablet_manager.cc:1397] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:53:22.986995 28079 tablet_bootstrap.cc:492] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: No bootstrap required, opened a new log
I20260619 06:53:22.987229 28079 ts_tablet_manager.cc:1397] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 06:53:22.989763 28080 raft_consensus.cc:348] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.989763 28078 raft_consensus.cc:348] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.990061 28078 raft_consensus.cc:374] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:22.990061 28080 raft_consensus.cc:374] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:22.990128 28080 raft_consensus.cc:729] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a374d7b1b3a4942a4f51f4a70ab73bd, State: Initialized, Role: FOLLOWER
I20260619 06:53:22.990135 28078 raft_consensus.cc:729] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19db378a96844d5fb39b974c531b7570, State: Initialized, Role: FOLLOWER
I20260619 06:53:22.990628 28078 consensus_queue.cc:260] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.990628 28080 consensus_queue.cc:260] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.991297 28079 raft_consensus.cc:348] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.991674 28061 heartbeater.cc:503] Master 127.22.113.62:40681 was elected leader, sending a full tablet report...
I20260619 06:53:22.991843 28079 raft_consensus.cc:374] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:22.991847 27799 heartbeater.cc:503] Master 127.22.113.62:40681 was elected leader, sending a full tablet report...
I20260619 06:53:22.991855 28080 ts_tablet_manager.cc:1428] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 06:53:22.991911 28079 raft_consensus.cc:729] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f429460a04b64d3dbab4fabf9c6d4914, State: Initialized, Role: FOLLOWER
I20260619 06:53:22.992271 28078 ts_tablet_manager.cc:1428] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 06:53:22.992424 28079 consensus_queue.cc:260] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:22.993577 28079 ts_tablet_manager.cc:1428] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260619 06:53:22.993929 27930 heartbeater.cc:503] Master 127.22.113.62:40681 was elected leader, sending a full tablet report...
W20260619 06:53:23.060932 27800 tablet.cc:2369] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:23.166872 28062 tablet.cc:2369] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:23.241973 27931 tablet.cc:2369] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:23.424829 28084 raft_consensus.cc:482] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:23.425100 28084 raft_consensus.cc:504] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:23.426335 28084 leader_election.cc:288] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f429460a04b64d3dbab4fabf9c6d4914 (127.22.113.2:39665), 9a374d7b1b3a4942a4f51f4a70ab73bd (127.22.113.3:37951)
I20260619 06:53:23.429036 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.426883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:53875 (local address 127.22.113.3:37951)
0619 06:53:23.427018 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:23.427023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:23.427204 (+ 181us) server_negotiation.cc:407] Connection header received
0619 06:53:23.427396 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:23.427399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:23.427455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:23.427548 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:23.428487 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:23.428491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:23.428494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:23.428506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:23.428531 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:23.428540 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:23.428649 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:23.428800 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:53:23.428861 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:53:23.429064 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.426695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37951 (local address 127.22.113.1:53875)
0619 06:53:23.427091 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:23.427106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:23.427228 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:23.427594 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:23.427597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:23.427614 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:23.428277 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:23.428280 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:23.428297 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:23.428335 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:23.428685 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:23.428689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:23.428692 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:53:23.428796 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:53:23.428843 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":95,"threads_started":1}
I20260619 06:53:23.429175 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.426778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:47749 (local address 127.22.113.2:39665)
0619 06:53:23.426937 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:23.426942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:23.427425 (+ 483us) server_negotiation.cc:407] Connection header received
0619 06:53:23.427591 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:23.427594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:23.427645 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:23.427740 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:23.428707 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:23.428711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:23.428716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:23.428733 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:23.428760 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:23.428773 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:23.428861 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:23.428979 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:53:23.429022 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:53:23.429394 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.426695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:39665 (local address 127.22.113.1:47749)
0619 06:53:23.427291 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:23.427311 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:23.427453 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:23.427775 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:23.427778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:23.427792 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:23.428522 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:23.428524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:23.428539 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:23.428571 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:23.428865 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:23.428868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:23.428871 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:53:23.428967 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:53:23.429004 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":30,"thread_start_us":70,"threads_started":1}
I20260619 06:53:23.429973 28015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c7bbad44d48deb92f904c88b6197a" candidate_uuid: "19db378a96844d5fb39b974c531b7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" is_pre_election: true
I20260619 06:53:23.430162 27884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c7bbad44d48deb92f904c88b6197a" candidate_uuid: "19db378a96844d5fb39b974c531b7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f429460a04b64d3dbab4fabf9c6d4914" is_pre_election: true
I20260619 06:53:23.430410 28015 raft_consensus.cc:2454] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19db378a96844d5fb39b974c531b7570 in term 0.
I20260619 06:53:23.430466 27884 raft_consensus.cc:2454] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19db378a96844d5fb39b974c531b7570 in term 0.
I20260619 06:53:23.430974 27688 leader_election.cc:302] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19db378a96844d5fb39b974c531b7570, 9a374d7b1b3a4942a4f51f4a70ab73bd, f429460a04b64d3dbab4fabf9c6d4914; no voters:
I20260619 06:53:23.431286 28084 raft_consensus.cc:2781] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:23.431375 28084 raft_consensus.cc:482] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:23.431419 28084 raft_consensus.cc:3037] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:23.432507 28084 raft_consensus.cc:504] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:23.432844 28084 leader_election.cc:288] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [CANDIDATE]: Term 1 election: Requested vote from peers f429460a04b64d3dbab4fabf9c6d4914 (127.22.113.2:39665), 9a374d7b1b3a4942a4f51f4a70ab73bd (127.22.113.3:37951)
I20260619 06:53:23.433216 27884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c7bbad44d48deb92f904c88b6197a" candidate_uuid: "19db378a96844d5fb39b974c531b7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f429460a04b64d3dbab4fabf9c6d4914"
I20260619 06:53:23.433319 28015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c7bbad44d48deb92f904c88b6197a" candidate_uuid: "19db378a96844d5fb39b974c531b7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd"
I20260619 06:53:23.433386 27884 raft_consensus.cc:3037] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:23.433495 28015 raft_consensus.cc:3037] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:23.434552 28015 raft_consensus.cc:2454] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19db378a96844d5fb39b974c531b7570 in term 1.
I20260619 06:53:23.434823 27884 raft_consensus.cc:2454] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19db378a96844d5fb39b974c531b7570 in term 1.
I20260619 06:53:23.434854 27689 leader_election.cc:302] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19db378a96844d5fb39b974c531b7570, 9a374d7b1b3a4942a4f51f4a70ab73bd; no voters:
I20260619 06:53:23.435082 28084 raft_consensus.cc:2781] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:23.435370 28084 raft_consensus.cc:686] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 1 LEADER]: Becoming Leader. State: Replica: 19db378a96844d5fb39b974c531b7570, State: Running, Role: LEADER
I20260619 06:53:23.435781 28084 consensus_queue.cc:237] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } }
I20260619 06:53:23.439636 27611 catalog_manager.cc:5495] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19db378a96844d5fb39b974c531b7570 (127.22.113.1). New cstate: current_term: 1 leader_uuid: "19db378a96844d5fb39b974c531b7570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19db378a96844d5fb39b974c531b7570" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:23.532997 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.531125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.22.113.1:46529)
0619 06:53:23.531403 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:23.531406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:23.531478 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:53:23.531681 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:23.531683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:23.531744 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:23.531831 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:23.532650 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:23.532652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:23.532654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:23.532662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:23.532678 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:23.532686 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:23.532757 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:23.532850 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:53:23.532883 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":126,"threads_started":1}
I20260619 06:53:23.540088 28015 raft_consensus.cc:1261] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 1 FOLLOWER]: Refusing update from remote peer 19db378a96844d5fb39b974c531b7570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:23.540088 27884 raft_consensus.cc:1261] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 1 FOLLOWER]: Refusing update from remote peer 19db378a96844d5fb39b974c531b7570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:23.540861 28084 consensus_queue.cc:1035] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a374d7b1b3a4942a4f51f4a70ab73bd" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:23.541044 28089 consensus_queue.cc:1035] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f429460a04b64d3dbab4fabf9c6d4914" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:23.551695 28098 mvcc.cc:205] Tried to move back new op lower bound from 7298465806490951680 to 7298465806075854848. Current Snapshot: MvccSnapshot[applied={T|T < 7298465806490951680}]
I20260619 06:53:23.562036 27611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:23.562227 27611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:23.565254 27611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ffda89361444f1e903d33200730718b: Sending DeleteTablet for 3 replicas of tablet ef6c7bbad44d48deb92f904c88b6197a
I20260619 06:53:23.566237 27864 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c7bbad44d48deb92f904c88b6197a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:23 UTC) from {username='slave'} at 127.0.0.1:59412
I20260619 06:53:23.566274 27733 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c7bbad44d48deb92f904c88b6197a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:23 UTC) from {username='slave'} at 127.0.0.1:41924
I20260619 06:53:23.566901 28107 tablet_replica.cc:330] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570: stopping tablet replica
I20260619 06:53:23.567188 28107 raft_consensus.cc:2229] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:23.567159 27995 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c7bbad44d48deb92f904c88b6197a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:23 UTC) from {username='slave'} at 127.0.0.1:58136
I20260619 06:53:23.567610 28107 raft_consensus.cc:2258] T ef6c7bbad44d48deb92f904c88b6197a P 19db378a96844d5fb39b974c531b7570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:23.567983 28108 tablet_replica.cc:330] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: stopping tablet replica
I20260619 06:53:23.568275 28108 raft_consensus.cc:2229] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:23.568573 28108 raft_consensus.cc:2258] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:23.569049 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27671
I20260619 06:53:23.569860 28106 tablet_replica.cc:330] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: stopping tablet replica
I20260619 06:53:23.569890 28108 ts_tablet_manager.cc:1905] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:23.570135 28106 raft_consensus.cc:2229] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:23.570396 28106 raft_consensus.cc:2258] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:23.572824 28106 ts_tablet_manager.cc:1905] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:23.573871 28108 ts_tablet_manager.cc:1918] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:23.573992 28108 log.cc:1196] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/ef6c7bbad44d48deb92f904c88b6197a
I20260619 06:53:23.574429 28108 ts_tablet_manager.cc:1939] T ef6c7bbad44d48deb92f904c88b6197a P 9a374d7b1b3a4942a4f51f4a70ab73bd: Deleting consensus metadata
I20260619 06:53:23.575675 28106 ts_tablet_manager.cc:1918] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:23.575801 28106 log.cc:1196] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/ef6c7bbad44d48deb92f904c88b6197a
I20260619 06:53:23.575964 27598 catalog_manager.cc:4841] TS 9a374d7b1b3a4942a4f51f4a70ab73bd (127.22.113.3:37951): tablet ef6c7bbad44d48deb92f904c88b6197a (table test-table [id=f0a6062765bd4d27ac71a2ce65c8914c]) successfully deleted
I20260619 06:53:23.576156 28106 ts_tablet_manager.cc:1939] T ef6c7bbad44d48deb92f904c88b6197a P f429460a04b64d3dbab4fabf9c6d4914: Deleting consensus metadata
W20260619 06:53:23.577416 27597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:46529 (error 108)
I20260619 06:53:23.577483 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27802
I20260619 06:53:23.577913 27597 catalog_manager.cc:4841] TS f429460a04b64d3dbab4fabf9c6d4914 (127.22.113.2:39665): tablet ef6c7bbad44d48deb92f904c88b6197a (table test-table [id=f0a6062765bd4d27ac71a2ce65c8914c]) successfully deleted
I20260619 06:53:23.580518 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.580041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46529 (local address 127.0.0.1:41950)
0619 06:53:23.580291 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:23.580371 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260619 06:53:23.580793 27597 catalog_manager.cc:4584] TS 19db378a96844d5fb39b974c531b7570 (127.22.113.1:46529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef6c7bbad44d48deb92f904c88b6197a: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46529: connect: Connection refused (error 111)
I20260619 06:53:23.584362 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27933
I20260619 06:53:23.590545 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 27580
2026-06-19T06:53:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1518 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:53:23.613899 22980 mini_oidc.cc:152] Starting JWKS server
I20260619 06:53:23.615864 22980 webserver.cc:460] Webserver started at https://127.0.0.1:37543/ using document root <none> and password file <none>
I20260619 06:53:23.616060 22980 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:53:23.616366 22980 webserver.cc:460] Webserver started at http://127.0.0.1:37755/ using document root <none> and password file <none>
2026-06-19T06:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:23Z Disabled control of system clock
I20260619 06:53:23.627815 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:45859
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:45859
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/testchainca.cert with env {}
W20260619 06:53:23.751184 28116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:23.751683 28116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:23.751772 28116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:23.758280 28116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:23.758394 28116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:23.758417 28116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:23.758486 28116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:23.758512 28116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:53:23.758541 28116 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:53:23.758575 28116 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37543/jwks.json
I20260619 06:53:23.765076 28116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:45859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:23.766383 28116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:23.767620 28116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:23.777374 28121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:23.777369 28122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:23.777697 28124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:23.777650 28116 server_base.cc:1034] running on GCE node
I20260619 06:53:23.778488 28116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:23.779907 28116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:23.781078 28116 hybrid_clock.cc:648] HybridClock initialized: now 1781852003781062 us; error 33 us; skew 500 ppm
I20260619 06:53:23.783313 28116 webserver.cc:460] Webserver started at http://127.22.113.62:45675/ using document root <none> and password file <none>
I20260619 06:53:23.784015 28116 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:23.784085 28116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:23.784325 28116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:23.786690 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "64ea3a31d0cc40179e0c5e8261c3349a"
format_stamp: "Formatted at 2026-06-19 06:53:23 on dist-test-slave-g8zg"
server_key: "dcd7888991671b2cb86f6444cc111c97"
server_key_iv: "96881cbeab702c846bc53e8904953aba"
server_key_version: "encryptionkey@0"
I20260619 06:53:23.787388 28116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "64ea3a31d0cc40179e0c5e8261c3349a"
format_stamp: "Formatted at 2026-06-19 06:53:23 on dist-test-slave-g8zg"
server_key: "dcd7888991671b2cb86f6444cc111c97"
server_key_iv: "96881cbeab702c846bc53e8904953aba"
server_key_version: "encryptionkey@0"
I20260619 06:53:23.791798 28116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:23.795122 28130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:23.796898 28116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:23.797058 28116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "64ea3a31d0cc40179e0c5e8261c3349a"
format_stamp: "Formatted at 2026-06-19 06:53:23 on dist-test-slave-g8zg"
server_key: "dcd7888991671b2cb86f6444cc111c97"
server_key_iv: "96881cbeab702c846bc53e8904953aba"
server_key_version: "encryptionkey@0"
I20260619 06:53:23.797200 28116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:23.811961 28116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:23.812762 28116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:23.812985 28116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:23.821674 28116 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:45859
I20260619 06:53:23.821712 28182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:45859 every 8 connection(s)
I20260619 06:53:23.822873 28116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:23.824911 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28116
I20260619 06:53:23.825054 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:23.825376 22980 external_mini_cluster.cc:1428] Setting key f6fda2a3bb4d310692454e6ee63b36bd
I20260619 06:53:23.827013 28183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:23.832743 28183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a: Bootstrap starting.
I20260619 06:53:23.834993 28183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:23.835801 28183 log.cc:824] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:23.836169 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.826825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.22.113.62:45859)
0619 06:53:23.827759 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:23.827779 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:23.827825 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:53:23.828633 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:23.828652 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:23.829125 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:23.829561 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:23.830052 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.831123 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.832199 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.832734 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.832915 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:23.833935 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:23.833941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:23.833957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:23.834437 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:23.834504 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:23.834526 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:23.834643 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:23.834780 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:53:23.835125 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":169,"threads_started":1}
I20260619 06:53:23.838100 28183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a: No bootstrap required, opened a new log
I20260619 06:53:23.839696 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:53:23.842110 28183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } }
I20260619 06:53:23.842348 28183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:23.842427 28183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ea3a31d0cc40179e0c5e8261c3349a, State: Initialized, Role: FOLLOWER
I20260619 06:53:23.842837 28183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } }
I20260619 06:53:23.842988 28183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:23.843071 28183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:23.843186 28183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:23.844276 28183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } }
I20260619 06:53:23.844611 28183 leader_election.cc:302] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64ea3a31d0cc40179e0c5e8261c3349a; no voters:
I20260619 06:53:23.844985 28183 leader_election.cc:288] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:23.845141 28188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:23.845481 28188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [term 1 LEADER]: Becoming Leader. State: Replica: 64ea3a31d0cc40179e0c5e8261c3349a, State: Running, Role: LEADER
I20260619 06:53:23.845892 28188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } }
I20260619 06:53:23.846310 28183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:23.847864 28190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } } }
I20260619 06:53:23.847869 28189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64ea3a31d0cc40179e0c5e8261c3349a. Latest consensus state: current_term: 1 leader_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ea3a31d0cc40179e0c5e8261c3349a" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 45859 } } }
I20260619 06:53:23.848027 28189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:23.848027 28190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:23.849114 28203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:23.852325 28203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:23.858780 28203 catalog_manager.cc:1347] Generated new cluster ID: 0c137c73d7b44fbc84c3380e9307c2b0
I20260619 06:53:23.858876 28203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:23.868395 28203 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:23.869418 28203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:23.882099 28203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64ea3a31d0cc40179e0c5e8261c3349a: Generated new TSK 0
I20260619 06:53:23.883049 28203 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:53:23.911942 28213 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:53:23.912082 28213 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:53:23.912643 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:23.894971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.22.113.62:45859)
0619 06:53:23.895124 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:23.895132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:23.895308 (+ 176us) server_negotiation.cc:407] Connection header received
0619 06:53:23.895593 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:23.895596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:23.895666 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:23.895809 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:53:23.896068 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.896922 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.897827 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.898180 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.898291 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:23.898639 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:53:23.912217 (+ 13578us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:53:23.912489 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 06:53:23.912726 28186 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:53:23.914068 28212 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:23.894838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:45859 (local address 127.0.0.1:53858)
0619 06:53:23.895181 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:23.895196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:23.895401 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:23.895824 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:23.895826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:23.895833 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:53:23.895954 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:23.895959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.897040 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.897042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:23.897708 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:23.897715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:23.898299 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:23.898301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:23.898384 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:23.898431 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:53:23.912474 (+ 14043us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:53:23.912704 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.113.62:45859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":279,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.113.62:45859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:53:23.914907 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28116
2026-06-19T06:53:23Z chronyd exiting
I20260619 06:53:23.953344 22980 test_util.cc:175] -----------------------------------------------
I20260619 06:53:23.953459 22980 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851976522130-22980-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:53:23.954649 22980 mini_oidc.cc:152] Starting JWKS server
I20260619 06:53:23.955663 22980 webserver.cc:460] Webserver started at https://127.0.0.1:32865/ using document root <none> and password file <none>
I20260619 06:53:23.955749 22980 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:53:23.955958 22980 webserver.cc:460] Webserver started at http://127.0.0.1:37011/ using document root <none> and password file <none>
2026-06-19T06:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:23Z Disabled control of system clock
I20260619 06:53:23.966614 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40405
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40405
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/testchainca.cert with env {}
W20260619 06:53:24.094405 28219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:24.094709 28219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:24.094753 28219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:24.098869 28219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:24.098943 28219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:24.098960 28219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:24.098978 28219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:24.098994 28219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:53:24.099010 28219 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:53:24.099025 28219 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32865/jwks.json
I20260619 06:53:24.104559 28219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:24.105676 28219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:24.106710 28219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:53:24.116434 28219 server_base.cc:1034] running on GCE node
W20260619 06:53:24.116401 28224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:24.116688 28225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:24.116287 28227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:24.118402 28219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:24.119715 28219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:24.120960 28219 hybrid_clock.cc:648] HybridClock initialized: now 1781852004120936 us; error 43 us; skew 500 ppm
I20260619 06:53:24.122999 28219 webserver.cc:460] Webserver started at http://127.22.113.62:37323/ using document root <none> and password file <none>
I20260619 06:53:24.123688 28219 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:24.123780 28219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:24.124028 28219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:24.126144 28219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "bfc05d3667dd473aa9d3f42a61d487e3"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "b4178a6c6c24e5667a670421e2200222"
server_key_iv: "244ab26238d24c1332ffe16beb1a24a5"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.126724 28219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "bfc05d3667dd473aa9d3f42a61d487e3"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "b4178a6c6c24e5667a670421e2200222"
server_key_iv: "244ab26238d24c1332ffe16beb1a24a5"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.130892 28219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:24.134056 28233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:24.135634 28219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:24.135815 28219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "bfc05d3667dd473aa9d3f42a61d487e3"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "b4178a6c6c24e5667a670421e2200222"
server_key_iv: "244ab26238d24c1332ffe16beb1a24a5"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.135993 28219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:24.161132 28219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:24.162000 28219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:24.162245 28219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:24.170632 28219 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40405
I20260619 06:53:24.170723 28285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40405 every 8 connection(s)
I20260619 06:53:24.171793 28219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:24.173743 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28219
I20260619 06:53:24.173847 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:24.174181 22980 external_mini_cluster.cc:1428] Setting key 9e3da046460ecf4c504d2e0bc80a2808
I20260619 06:53:24.176295 28286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:24.184206 28286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3: Bootstrap starting.
I20260619 06:53:24.185882 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.175811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.22.113.62:40405)
0619 06:53:24.176603 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.176627 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.176679 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:53:24.177674 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.177699 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.178243 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.178673 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:24.179184 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.180777 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.181740 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.182466 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.182651 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.183589 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:24.183596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:24.183612 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:24.184149 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:24.184229 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:24.184258 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:24.184378 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:24.184561 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:53:24.184855 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260619 06:53:24.187281 28286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:24.188233 28286 log.cc:824] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:24.189522 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:53:24.190773 28286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3: No bootstrap required, opened a new log
I20260619 06:53:24.194691 28286 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } }
I20260619 06:53:24.194938 28286 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:24.195026 28286 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc05d3667dd473aa9d3f42a61d487e3, State: Initialized, Role: FOLLOWER
I20260619 06:53:24.195566 28286 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } }
I20260619 06:53:24.195700 28286 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:24.195770 28286 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:24.195909 28286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:24.196918 28286 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } }
I20260619 06:53:24.197288 28286 leader_election.cc:302] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfc05d3667dd473aa9d3f42a61d487e3; no voters:
I20260619 06:53:24.197619 28286 leader_election.cc:288] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:24.197814 28291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:24.198135 28291 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [term 1 LEADER]: Becoming Leader. State: Replica: bfc05d3667dd473aa9d3f42a61d487e3, State: Running, Role: LEADER
I20260619 06:53:24.198535 28291 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } }
I20260619 06:53:24.199107 28286 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:24.200601 28293 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc05d3667dd473aa9d3f42a61d487e3. Latest consensus state: current_term: 1 leader_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } } }
I20260619 06:53:24.200644 28292 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc05d3667dd473aa9d3f42a61d487e3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40405 } } }
I20260619 06:53:24.200752 28293 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.200763 28292 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.201251 28300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:24.204735 28300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:24.211328 28300 catalog_manager.cc:1347] Generated new cluster ID: c5be5d756add473bab883dcd67ec55dc
I20260619 06:53:24.211400 28300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:24.223448 28300 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:24.224474 28300 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:24.229914 28300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfc05d3667dd473aa9d3f42a61d487e3: Generated new TSK 0
I20260619 06:53:24.230757 28300 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:53:24.265841 28316 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:53:24.265962 28316 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:53:24.266681 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.250199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.22.113.62:40405)
0619 06:53:24.250364 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.250369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.250539 (+ 170us) server_negotiation.cc:407] Connection header received
0619 06:53:24.250701 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.250705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.250770 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.250908 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:53:24.251143 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.252025 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.253019 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.253395 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.253517 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.253885 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:53:24.266221 (+ 12336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:53:24.266507 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:53:24.266794 28289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 06:53:24.266861 28315 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:24.250082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40405 (local address 127.0.0.1:36452)
0619 06:53:24.250436 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:24.250450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:24.250566 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:24.250913 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:24.250915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:24.250922 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:53:24.251035 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.251039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.252150 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.252152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.252882 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.252892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.253559 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.253563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.253646 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.253675 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:53:24.266488 (+ 12813us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:53:24.266659 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.113.62:40405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32865/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
I20260619 06:53:24.268131 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28219
2026-06-19T06:53:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (354 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:53:24.309374 22980 mini_oidc.cc:152] Starting JWKS server
I20260619 06:53:24.310343 22980 webserver.cc:460] Webserver started at https://127.0.0.1:33039/ using document root <none> and password file <none>
I20260619 06:53:24.310479 22980 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:53:24.310799 22980 webserver.cc:460] Webserver started at http://127.0.0.1:34163/ using document root <none> and password file <none>
2026-06-19T06:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:24Z Disabled control of system clock
I20260619 06:53:24.320956 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:33053
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:33053
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:53:24.435467 28322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:24.435766 28322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:24.435808 28322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:24.439731 28322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:24.439795 28322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:24.439810 28322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:24.439826 28322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:24.439841 28322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:53:24.439857 28322 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:53:24.439872 28322 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33039/jwks.json
I20260619 06:53:24.444532 28322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:33053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:24.445664 28322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:24.446946 28322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:24.456349 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:24.456338 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:24.456617 28322 server_base.cc:1034] running on GCE node
W20260619 06:53:24.456349 28330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:24.457253 28322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:24.458315 28322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:24.459510 28322 hybrid_clock.cc:648] HybridClock initialized: now 1781852004459444 us; error 52 us; skew 500 ppm
I20260619 06:53:24.461413 28322 webserver.cc:460] Webserver started at http://127.22.113.62:36021/ using document root <none> and password file <none>
I20260619 06:53:24.461997 28322 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:24.462061 28322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:24.462275 28322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:24.464368 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "a6a94dc8694547c98afb67d65dba96ed"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "872b77ea0c508924e972a6d4e5150dca"
server_key_iv: "c1d6401e50eba954cd3c5f74462b8b3c"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.464962 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "a6a94dc8694547c98afb67d65dba96ed"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "872b77ea0c508924e972a6d4e5150dca"
server_key_iv: "c1d6401e50eba954cd3c5f74462b8b3c"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.468708 28322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:24.471177 28336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:24.472630 28322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:24.472800 28322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "a6a94dc8694547c98afb67d65dba96ed"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "872b77ea0c508924e972a6d4e5150dca"
server_key_iv: "c1d6401e50eba954cd3c5f74462b8b3c"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.472926 28322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:24.482733 28322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:24.483503 28322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:24.483711 28322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:24.491739 28322 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:33053
I20260619 06:53:24.491827 28388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:33053 every 8 connection(s)
I20260619 06:53:24.492816 28322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:24.496865 28389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:24.497560 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28322
I20260619 06:53:24.497722 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:24.498078 22980 external_mini_cluster.cc:1428] Setting key ad015dc0267aa30ec3588cfecf3f27e0
I20260619 06:53:24.504115 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed: Bootstrap starting.
I20260619 06:53:24.506733 28389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:24.507687 28389 log.cc:824] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:24.508961 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.499720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.22.113.62:33053)
0619 06:53:24.500544 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.500562 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.500604 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:53:24.501504 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.501524 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.501989 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.502369 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:24.502842 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.504002 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.505026 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.505557 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.505751 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.506710 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:24.506715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:24.506729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:24.507230 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:24.507363 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:24.507386 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:24.507512 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:24.507639 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 06:53:24.507918 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":164,"threads_started":1}
I20260619 06:53:24.510377 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed: No bootstrap required, opened a new log
I20260619 06:53:24.512466 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:53:24.514886 28389 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } }
I20260619 06:53:24.515161 28389 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:24.515247 28389 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a94dc8694547c98afb67d65dba96ed, State: Initialized, Role: FOLLOWER
I20260619 06:53:24.515779 28389 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } }
I20260619 06:53:24.515937 28389 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:24.516034 28389 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:24.516144 28389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:24.517247 28389 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } }
I20260619 06:53:24.517633 28389 leader_election.cc:302] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a94dc8694547c98afb67d65dba96ed; no voters:
I20260619 06:53:24.518019 28389 leader_election.cc:288] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:24.518239 28394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:24.518602 28394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [term 1 LEADER]: Becoming Leader. State: Replica: a6a94dc8694547c98afb67d65dba96ed, State: Running, Role: LEADER
I20260619 06:53:24.518966 28394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } }
I20260619 06:53:24.519572 28389 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:24.520903 28396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a94dc8694547c98afb67d65dba96ed. Latest consensus state: current_term: 1 leader_uuid: "a6a94dc8694547c98afb67d65dba96ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } } }
I20260619 06:53:24.521060 28396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.521468 28395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a94dc8694547c98afb67d65dba96ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a94dc8694547c98afb67d65dba96ed" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33053 } } }
I20260619 06:53:24.521595 28395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.522864 28403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:24.526240 28403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:24.533128 28403 catalog_manager.cc:1347] Generated new cluster ID: 30516e7ad7c140e98bb0966fbfcad1f1
I20260619 06:53:24.533244 28403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:24.545478 28403 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:24.546665 28403 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:24.552589 28403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a94dc8694547c98afb67d65dba96ed: Generated new TSK 0
I20260619 06:53:24.553462 28403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:24.573310 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.567846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.22.113.62:33053)
0619 06:53:24.567995 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.568000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.568172 (+ 172us) server_negotiation.cc:407] Connection header received
0619 06:53:24.568367 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.568369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.568432 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.568537 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:53:24.568852 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.569423 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.570701 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.570945 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.571068 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.571484 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:53:24.572882 (+ 1398us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:53:24.573164 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 06:53:24.573410 28392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 06:53:24.576121 28418 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:24.567672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:33053 (local address 127.0.0.1:49694)
0619 06:53:24.568056 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:24.568073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:24.568211 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:24.568565 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:24.568568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:24.568577 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:53:24.568719 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.568726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.569539 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.569543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.570573 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.570583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.571118 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.571122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.571211 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.571289 (+ 78us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:53:24.573054 (+ 1765us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:53:24.575954 (+ 2900us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.113.62:33053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":2697,"thread_start_us":124,"threads_started":1}
I20260619 06:53:24.576593 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28322
E20260619 06:53:24.585512 28419 webserver.cc:533] Webserver: sslize failed:
E20260619 06:53:24.585603 28419 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T06:53:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:24Z Disabled control of system clock
I20260619 06:53:24.626704 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:37701
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:37701
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/testchainca.cert with env {}
W20260619 06:53:24.735497 28423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:24.735769 28423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:24.735809 28423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:24.739823 28423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:24.739892 28423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:24.739907 28423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:24.739924 28423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:24.739940 28423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:53:24.739956 28423 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:53:24.740024 28423 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:53:24.745486 28423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:37701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:37701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:24.746623 28423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:24.747639 28423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:53:24.756770 28423 server_base.cc:1034] running on GCE node
W20260619 06:53:24.756731 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:24.756688 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:24.756707 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:24.757766 28423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:24.758818 28423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:24.760022 28423 hybrid_clock.cc:648] HybridClock initialized: now 1781852004759952 us; error 82 us; skew 500 ppm
I20260619 06:53:24.762027 28423 webserver.cc:460] Webserver started at http://127.22.113.62:34107/ using document root <none> and password file <none>
I20260619 06:53:24.762633 28423 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:24.762697 28423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:24.762924 28423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:24.765000 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "3990c3f5294f4c429eb70d2f407472b7"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "c17fc3cdeb25cd0046dae1181c31cf92"
server_key_iv: "b67a477a05edf6952cd4b7796b153947"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.765573 28423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "3990c3f5294f4c429eb70d2f407472b7"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "c17fc3cdeb25cd0046dae1181c31cf92"
server_key_iv: "b67a477a05edf6952cd4b7796b153947"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.769457 28423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:24.772181 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:24.773594 28423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:24.773757 28423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "3990c3f5294f4c429eb70d2f407472b7"
format_stamp: "Formatted at 2026-06-19 06:53:24 on dist-test-slave-g8zg"
server_key: "c17fc3cdeb25cd0046dae1181c31cf92"
server_key_iv: "b67a477a05edf6952cd4b7796b153947"
server_key_version: "encryptionkey@0"
I20260619 06:53:24.773877 28423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:24.797184 28423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:24.797921 28423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:24.798130 28423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:24.806447 28423 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:37701
I20260619 06:53:24.806529 28489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:37701 every 8 connection(s)
I20260619 06:53:24.807482 28423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:24.811062 28490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:24.813558 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28423
I20260619 06:53:24.813719 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:24.814060 22980 external_mini_cluster.cc:1428] Setting key eb55e9e7c10fe72a6cf0cb32361be5b8
I20260619 06:53:24.818585 28490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7: Bootstrap starting.
I20260619 06:53:24.821558 28490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:24.822595 28490 log.cc:824] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:24.825328 28490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7: No bootstrap required, opened a new log
I20260619 06:53:24.826090 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.815748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.22.113.62:37701)
0619 06:53:24.816688 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.816709 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.816752 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:53:24.817766 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.817789 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.818278 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.818660 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:24.819150 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.820325 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.821807 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.822473 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.822612 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.823942 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:24.823947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:24.823963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:24.824396 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:24.824456 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:24.824489 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:24.824580 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:24.824761 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 06:53:24.825105 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":179,"threads_started":1}
I20260619 06:53:24.828917 28490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } }
I20260619 06:53:24.829190 28490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:24.829286 28490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3990c3f5294f4c429eb70d2f407472b7, State: Initialized, Role: FOLLOWER
I20260619 06:53:24.829792 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:53:24.829838 28490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } }
I20260619 06:53:24.830004 28490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:24.830077 28490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:24.830176 28490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:24.831564 28490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } }
I20260619 06:53:24.831997 28490 leader_election.cc:302] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3990c3f5294f4c429eb70d2f407472b7; no voters:
I20260619 06:53:24.832388 28490 leader_election.cc:288] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:24.832556 28495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:24.832868 28495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3990c3f5294f4c429eb70d2f407472b7, State: Running, Role: LEADER
I20260619 06:53:24.833285 28495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } }
I20260619 06:53:24.833866 28490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:24.835156 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3990c3f5294f4c429eb70d2f407472b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } } }
I20260619 06:53:24.835350 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.835671 28497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3990c3f5294f4c429eb70d2f407472b7. Latest consensus state: current_term: 1 leader_uuid: "3990c3f5294f4c429eb70d2f407472b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990c3f5294f4c429eb70d2f407472b7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37701 } } }
I20260619 06:53:24.835774 28497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:24.835881 28504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:24.838946 28504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:24.845881 28504 catalog_manager.cc:1347] Generated new cluster ID: da0cf7483aec449ebac4b14c63c105d5
I20260619 06:53:24.845959 28504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:24.875764 28504 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:24.877130 28504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:24.890691 28504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3990c3f5294f4c429eb70d2f407472b7: Generated new TSK 0
I20260619 06:53:24.891904 28504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:25.366657 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:24.902292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.22.113.62:37701)
0619 06:53:24.902460 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:24.902465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:24.902622 (+ 157us) server_negotiation.cc:407] Connection header received
0619 06:53:24.902788 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:24.902792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:24.902869 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:24.903036 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:53:24.903352 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.904300 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.905287 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.905660 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.905769 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.906096 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:53:25.366217 (+460121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:53:25.366526 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56210: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:53:25.366768 28493 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56210: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:53:25.367108 28519 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:24.902179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:37701 (local address 127.0.0.1:56210)
0619 06:53:24.902522 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:24.902536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:24.902653 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:24.903035 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:24.903038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:24.903044 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:53:24.903173 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.903178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.904439 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.904442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.905139 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:24.905147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:24.905792 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:24.905795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:24.905858 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:24.905885 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:53:25.366548 (+460663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:53:25.366863 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.113.62:37701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":137,"threads_started":1}
I20260619 06:53:25.367915 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28423
2026-06-19T06:53:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (771 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:25 dist-test-slave-g8zg krb5kdc[28531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:25 dist-test-slave-g8zg krb5kdc[28531](info): set up 2 sockets
Jun 19 06:53:25 dist-test-slave-g8zg krb5kdc[28531](info): commencing operation
krb5kdc: starting...
W20260619 06:53:26.675932 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:26 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:26.811611 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38457
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:26.934921 28544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:26.935360 28544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:26.935477 28544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:26.939697 28544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:26.939796 28544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:26.939833 28544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:26.939872 28544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:26.939905 28544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:26.945452 28544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:38457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:26.946704 28544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:26.947912 28544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:26.957864 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:26.957963 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:26.957791 28552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:26.958180 28544 server_base.cc:1034] running on GCE node
I20260619 06:53:26.958951 28544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:26.960244 28544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:26.961418 28544 hybrid_clock.cc:648] HybridClock initialized: now 1781852006961386 us; error 52 us; skew 500 ppm
Jun 19 06:53:26 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852006, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:26.967598 28544 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:26.968597 28544 webserver.cc:460] Webserver started at http://127.22.113.62:43777/ using document root <none> and password file <none>
I20260619 06:53:26.969154 28544 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:26.969216 28544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:26.969388 28544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:26.971580 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "7d1249c4ab924ecb9b50177b6bfcb395"
format_stamp: "Formatted at 2026-06-19 06:53:26 on dist-test-slave-g8zg"
server_key: "7ad5d63761c39a57c4e7f490e8810699"
server_key_iv: "5517f98d351f4d9c03679bf4ae0994b5"
server_key_version: "encryptionkey@0"
I20260619 06:53:26.972103 28544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "7d1249c4ab924ecb9b50177b6bfcb395"
format_stamp: "Formatted at 2026-06-19 06:53:26 on dist-test-slave-g8zg"
server_key: "7ad5d63761c39a57c4e7f490e8810699"
server_key_iv: "5517f98d351f4d9c03679bf4ae0994b5"
server_key_version: "encryptionkey@0"
I20260619 06:53:26.976096 28544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:26.978749 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:26.980448 28544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:53:26.980621 28544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "7d1249c4ab924ecb9b50177b6bfcb395"
format_stamp: "Formatted at 2026-06-19 06:53:26 on dist-test-slave-g8zg"
server_key: "7ad5d63761c39a57c4e7f490e8810699"
server_key_iv: "5517f98d351f4d9c03679bf4ae0994b5"
server_key_version: "encryptionkey@0"
I20260619 06:53:26.980746 28544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:27.004395 28544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:27.007995 28544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:27.008220 28544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:27.016496 28544 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:38457
I20260619 06:53:27.016528 28611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:38457 every 8 connection(s)
I20260619 06:53:27.017530 28544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:27.019146 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28544
I20260619 06:53:27.019325 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:27.019685 22980 external_mini_cluster.cc:1428] Setting key 50fffc1d4be9b07deecddebac2ab2cb3
I20260619 06:53:27.021481 28612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:27.028753 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395: Bootstrap starting.
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:27.032006 28612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:27.033000 28612 log.cc:824] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:27.035868 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395: No bootstrap required, opened a new log
I20260619 06:53:27.036233 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.021369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.22.113.62:38457)
0619 06:53:27.021926 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.021942 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.021981 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:53:27.022696 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.022714 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.023078 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.023501 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.024280 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.025296 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.026475 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.027005 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.027181 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.030578 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.030584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.030599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.030631 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.032662 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.033306 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.033311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.033316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.033472 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.033829 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.033832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.033833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.034344 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.034518 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.034915 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.035253 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":205,"threads_started":1}
I20260619 06:53:27.039921 28612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } }
I20260619 06:53:27.040175 28612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:27.040251 28612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1249c4ab924ecb9b50177b6bfcb395, State: Initialized, Role: FOLLOWER
I20260619 06:53:27.040758 28612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } }
I20260619 06:53:27.040887 28612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:27.040974 28612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:27.041110 28612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:27.042177 28612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } }
I20260619 06:53:27.042570 28612 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1249c4ab924ecb9b50177b6bfcb395; no voters:
I20260619 06:53:27.042941 28612 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:27.043144 28617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:27.043524 28617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [term 1 LEADER]: Becoming Leader. State: Replica: 7d1249c4ab924ecb9b50177b6bfcb395, State: Running, Role: LEADER
I20260619 06:53:27.043938 28617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } }
I20260619 06:53:27.044548 28612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:27.045924 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1249c4ab924ecb9b50177b6bfcb395. Latest consensus state: current_term: 1 leader_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } } }
I20260619 06:53:27.046073 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:27.046428 28618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1249c4ab924ecb9b50177b6bfcb395" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38457 } } }
I20260619 06:53:27.046530 28618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:27.046705 28626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:27.050371 28626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:27.056931 28626 catalog_manager.cc:1347] Generated new cluster ID: 16e499dab2ed4455a55079bdd26f77f0
I20260619 06:53:27.057015 28626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:27.068245 28626 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:27.069314 28626 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:27.079020 28626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1249c4ab924ecb9b50177b6bfcb395: Generated new TSK 0
I20260619 06:53:27.079828 28626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:27.115017 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:27.242265 28638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:27.242522 28638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:27.242566 28638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:27.246819 28638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:27.246915 28638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:27.246979 28638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:27.252494 28638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:27.253526 28638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:27.254518 28638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:27.264885 28643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:27.264914 28644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:27.265092 28638 server_base.cc:1034] running on GCE node
W20260619 06:53:27.264885 28646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:27.265760 28638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:27.266433 28638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:27.267602 28638 hybrid_clock.cc:648] HybridClock initialized: now 1781852007267561 us; error 58 us; skew 500 ppm
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.270537 28638 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:27.271538 28638 webserver.cc:460] Webserver started at http://127.22.113.1:41743/ using document root <none> and password file <none>
I20260619 06:53:27.272135 28638 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:27.272202 28638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:27.272418 28638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:27.274725 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "74b64b3426d64751aaf7d73cb63f10d2"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "8cd402853842170c473f1afd90b74251"
server_key_iv: "e9f34166e249a41ed4712a4216585f71"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.275516 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "74b64b3426d64751aaf7d73cb63f10d2"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "8cd402853842170c473f1afd90b74251"
server_key_iv: "e9f34166e249a41ed4712a4216585f71"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.282020 28638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260619 06:53:27.285775 28653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.287524 28638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:27.287689 28638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "74b64b3426d64751aaf7d73cb63f10d2"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "8cd402853842170c473f1afd90b74251"
server_key_iv: "e9f34166e249a41ed4712a4216585f71"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.287835 28638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:27.312687 28638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:27.316948 28638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:27.317171 28638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:27.317979 28638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:27.319209 28638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:27.319312 28638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.319407 28638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:27.319453 28638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.333525 28638 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:45933
I20260619 06:53:27.333624 28766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:45933 every 8 connection(s)
I20260619 06:53:27.334988 28638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:27.344064 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28638
I20260619 06:53:27.344254 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:27.344641 22980 external_mini_cluster.cc:1428] Setting key a6fe28af12683d266d1530d7ba9d687b
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:27.353670 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.337629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:46601 (local address 127.22.113.62:38457)
0619 06:53:27.337789 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.337793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.339041 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 06:53:27.340706 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.340710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.340778 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.340865 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.342275 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.342883 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.344492 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.344840 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.344962 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.349444 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.349448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.349450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.349478 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.351088 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.351724 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.351728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.351730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.351816 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.352202 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.352205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.352207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.352518 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.352631 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.353276 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.353452 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:53:27.354753 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.338194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38457 (local address 127.22.113.1:46601)
0619 06:53:27.338828 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:27.338882 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:27.340452 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:27.341076 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:27.341083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:27.341520 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:27.342108 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.342123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.343024 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.343030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.344335 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.344347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.345242 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.345249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.345658 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.346591 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:27.346619 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:27.349221 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:27.351263 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.351272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.351293 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.351606 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.351927 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.351931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.351933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.352095 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.352629 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:27.352637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:27.353006 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 06:53:27.353225 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 06:53:27.353689 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":182,"threads_started":1}
I20260619 06:53:27.356225 28767 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38457
I20260619 06:53:27.356539 28767 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:27.357162 28767 heartbeater.cc:511] Master 127.22.113.62:38457 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:27.359038 28576 ts_manager.cc:194] Registered new tserver with Master: 74b64b3426d64751aaf7d73cb63f10d2 (127.22.113.1:45933)
I20260619 06:53:27.360653 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:46601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:27.379791 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:27.510679 28772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:27.511024 28772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:27.511107 28772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:27.515512 28772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:27.515626 28772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:27.515759 28772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:27.521272 28772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:27.522543 28772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:27.523929 28772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:27.535367 28778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:27.535398 28780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:27.535784 28777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:27.536257 28772 server_base.cc:1034] running on GCE node
I20260619 06:53:27.536916 28772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:27.537837 28772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:27.539040 28772 hybrid_clock.cc:648] HybridClock initialized: now 1781852007539018 us; error 47 us; skew 500 ppm
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.542232 28772 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:27.543236 28772 webserver.cc:460] Webserver started at http://127.22.113.2:44861/ using document root <none> and password file <none>
I20260619 06:53:27.543910 28772 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:27.544004 28772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:27.544235 28772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:27.546685 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "accc104000004b82ac28d4856f11e422"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "1bb2eb6e1a648f322ed4048d4bed860d"
server_key_iv: "eada63ccb7812ed5420fcbbd4d7ca5d5"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.547356 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "accc104000004b82ac28d4856f11e422"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "1bb2eb6e1a648f322ed4048d4bed860d"
server_key_iv: "eada63ccb7812ed5420fcbbd4d7ca5d5"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.551498 28772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:53:27.554188 28787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.555691 28772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:27.555835 28772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "accc104000004b82ac28d4856f11e422"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "1bb2eb6e1a648f322ed4048d4bed860d"
server_key_iv: "eada63ccb7812ed5420fcbbd4d7ca5d5"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.555951 28772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:27.585830 28772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:27.590240 28772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:27.590461 28772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:27.591198 28772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:27.592262 28772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:27.592334 28772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.592396 28772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:27.592440 28772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.605535 28772 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:45645
I20260619 06:53:27.605552 28900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:45645 every 8 connection(s)
I20260619 06:53:27.606890 28772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:27.609414 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28772
I20260619 06:53:27.609535 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:27.609871 22980 external_mini_cluster.cc:1428] Setting key 3198c144304ea51804fe2ea761c7ac27
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:53:27.625726 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.611124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:59223 (local address 127.22.113.62:38457)
0619 06:53:27.611306 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.611311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.611329 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:27.612471 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.612477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.612572 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.612677 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.614622 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.615501 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.617563 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.617963 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.618089 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.621554 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.621559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.621561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.621593 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.623211 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.623889 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.623894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.623896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.623985 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.624379 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.624385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.624390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.624610 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.624718 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.625395 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.625536 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:53:27.626830 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.609928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38457 (local address 127.22.113.2:59223)
0619 06:53:27.610660 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:27.610710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:27.612240 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:27.613086 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:27.613099 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:27.613738 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:27.614463 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.614479 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.615688 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.615694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.617388 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.617403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.618302 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.618306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.618563 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.619484 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:27.619508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:27.621371 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:27.623423 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.623429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.623473 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.623767 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.624084 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.624087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.624089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.624243 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.624725 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:27.624734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:27.625112 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 06:53:27.625350 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 06:53:27.625732 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":196,"threads_started":1}
I20260619 06:53:27.628206 28901 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38457
I20260619 06:53:27.628557 28901 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:27.629276 28901 heartbeater.cc:511] Master 127.22.113.62:38457 requested a full tablet report, sending...
I20260619 06:53:27.630532 28576 ts_manager.cc:194] Registered new tserver with Master: accc104000004b82ac28d4856f11e422 (127.22.113.2:45645)
I20260619 06:53:27.631178 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:59223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:27.642980 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:27.760339 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:27.760630 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:27.760699 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:27.764969 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:27.765058 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:27.765153 28906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:27.770550 28906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:27.771775 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:27.772828 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:27.782907 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:27.782939 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:27.782907 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:27.783206 28906 server_base.cc:1034] running on GCE node
I20260619 06:53:27.783932 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:27.784849 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:27.786055 28906 hybrid_clock.cc:648] HybridClock initialized: now 1781852007786020 us; error 57 us; skew 500 ppm
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.789333 28906 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:27.790509 28906 webserver.cc:460] Webserver started at http://127.22.113.3:42837/ using document root <none> and password file <none>
I20260619 06:53:27.791188 28906 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:27.791309 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:27.791587 28906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:27.794201 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "d7220658c1fe4af69581a40b675b4d9d"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "37106166ba9cd0671cb469db893f6cac"
server_key_iv: "b5c594a043beed5c027ab124eea13e66"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.794860 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "d7220658c1fe4af69581a40b675b4d9d"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "37106166ba9cd0671cb469db893f6cac"
server_key_iv: "b5c594a043beed5c027ab124eea13e66"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.799621 28906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:53:27.802628 28921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.804299 28906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:27.804421 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "d7220658c1fe4af69581a40b675b4d9d"
format_stamp: "Formatted at 2026-06-19 06:53:27 on dist-test-slave-g8zg"
server_key: "37106166ba9cd0671cb469db893f6cac"
server_key_iv: "b5c594a043beed5c027ab124eea13e66"
server_key_version: "encryptionkey@0"
I20260619 06:53:27.804572 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:27.816622 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:27.820205 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:27.820425 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:27.821100 28906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:27.822062 28906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:27.822114 28906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.822157 28906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:27.822218 28906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:27.832741 28906 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40483
I20260619 06:53:27.832876 29034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40483 every 8 connection(s)
I20260619 06:53:27.833851 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:27.840695 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 28906
I20260619 06:53:27.840809 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:27.841202 22980 external_mini_cluster.cc:1428] Setting key 1d3a4b4c90b6fa4d369e43f1a3154686
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:27.849190 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.836064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:43025 (local address 127.22.113.62:38457)
0619 06:53:27.836273 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.836279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.837285 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 06:53:27.838665 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.838668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.838725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.838815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.840204 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.840827 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.842240 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.842492 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.842615 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.845416 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.845420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.845422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.845457 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.847206 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.847762 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.847765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.847766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.847844 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.848151 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.848153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.848154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.848358 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.848457 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.848932 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.849046 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:53:27.850049 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.836555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38457 (local address 127.22.113.3:43025)
0619 06:53:27.837105 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:27.837152 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:27.838441 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:27.839055 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:27.839062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:27.839534 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:27.840052 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.840065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.840976 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.840981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.842088 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:27.842098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.842667 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.842672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:27.842838 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.843535 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:27.843556 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:27.845188 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:27.847384 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.847388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.847399 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.847650 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.847930 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:27.847932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:27.847934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:27.848062 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:27.848457 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:27.848462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:27.848709 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 06:53:27.848906 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 06:53:27.849175 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260619 06:53:27.851516 29035 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38457
I20260619 06:53:27.851775 29035 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:27.852268 29035 heartbeater.cc:511] Master 127.22.113.62:38457 requested a full tablet report, sending...
I20260619 06:53:27.853446 28576 ts_manager.cc:194] Registered new tserver with Master: d7220658c1fe4af69581a40b675b4d9d (127.22.113.3:40483)
I20260619 06:53:27.854082 28576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:43025
I20260619 06:53:27.855779 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/test-admin.keytab.
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.872526 22980 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:53:27.875639 29042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:27.873975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.22.113.1:45933)
0619 06:53:27.874347 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.874355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.874385 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:53:27.874465 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.874471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.874717 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.874864 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.875423 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43092: BlockingRecv error: recv got EOF from 127.0.0.1:43092 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
W20260619 06:53:27.875790 22980 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.876991 22980 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.878029 22980 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:27.888936 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.879065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.22.113.1:45933)
0619 06:53:27.879185 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.879189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.879345 (+ 156us) server_negotiation.cc:407] Connection header received
0619 06:53:27.879485 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.879487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.879531 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.879606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.880237 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.881017 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.881896 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.882295 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.882383 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.885148 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.885158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.885173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.885197 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.887077 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.887580 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.887586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.887587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.887669 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.887933 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.887935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.887936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.888198 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.888446 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.888648 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.888792 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:53:27.893806 22980 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc' of type FILE
W20260619 06:53:27.893841 29042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:27.892546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.22.113.1:45933)
0619 06:53:27.892736 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.892740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.892816 (+ 76us) server_negotiation.cc:407] Connection header received
0619 06:53:27.893023 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.893027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.893080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.893176 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.893679 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43102: BlockingRecv error: recv got EOF from 127.0.0.1:43102 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.894951 22980 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.895978 22980 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:27.905624 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.897053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.22.113.1:45933)
0619 06:53:27.897205 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.897210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.897327 (+ 117us) server_negotiation.cc:407] Connection header received
0619 06:53:27.897495 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.897499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.897544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.897641 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.898160 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.898710 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.899600 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.899860 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.899983 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.902320 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.902324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.902327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.902360 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.903943 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.904371 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.904375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.904377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.904459 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.904737 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.904740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.904742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.904929 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.905104 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.905321 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.905438 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:53:27.908776 29042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:27.907769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.22.113.1:45933)
0619 06:53:27.907886 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.907892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.908044 (+ 152us) server_negotiation.cc:407] Connection header received
0619 06:53:27.908142 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.908144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.908180 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.908250 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.908640 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43124: BlockingRecv error: recv got EOF from 127.0.0.1:43124 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 06:53:27.909337 22980 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851976522130-22980-0/krb5kdc/krb5cc)
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.910496 22980 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:27.911499 22980 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:53:27 dist-test-slave-g8zg krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:53:27.920785 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:27.912419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.22.113.1:45933)
0619 06:53:27.912553 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:27.912557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:27.912673 (+ 116us) server_negotiation.cc:407] Connection header received
0619 06:53:27.912820 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:27.912824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:27.912871 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:27.912966 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:27.913498 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.914024 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.914829 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:27.915055 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:27.915175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:27.917607 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:27.917611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:27.917613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:27.917649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:27.919028 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.919536 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.919539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.919540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.919619 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:27.919947 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:27.919950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:27.919951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:27.920202 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:27.920358 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:27.920535 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:53:27.920649 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:53:27.922065 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28638
I20260619 06:53:27.928402 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28772
I20260619 06:53:27.934224 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28906
I20260619 06:53:27.940578 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 28544
2026-06-19T06:53:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2316 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:28 dist-test-slave-g8zg krb5kdc[29055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:28 dist-test-slave-g8zg krb5kdc[29055](info): set up 2 sockets
Jun 19 06:53:28 dist-test-slave-g8zg krb5kdc[29055](info): commencing operation
krb5kdc: starting...
W20260619 06:53:28.995843 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab.
Entry for principal oryx/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab.
I20260619 06:53:29.130782 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34219
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab
--principal=oryx/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:29.251688 29068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:29.251989 29068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:29.252032 29068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:29.256170 29068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:29.256242 29068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:29.256263 29068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:29.256280 29068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:29.256297 29068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:29.261802 29068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab
--principal=oryx/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:29.262877 29068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:29.263945 29068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:29.273398 29074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:29.273658 29068 server_base.cc:1034] running on GCE node
W20260619 06:53:29.273444 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:29.273882 29076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:29.274470 29068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:29.275874 29068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:29.277163 29068 hybrid_clock.cc:648] HybridClock initialized: now 1781852009277122 us; error 69 us; skew 500 ppm
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:29.280990 29068 init.cc:377] Logged in from keytab as oryx/127.22.113.62@KRBTEST.COM (short username oryx)
I20260619 06:53:29.282073 29068 webserver.cc:460] Webserver started at http://127.22.113.62:46537/ using document root <none> and password file <none>
I20260619 06:53:29.282716 29068 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:29.282811 29068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:29.283039 29068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:29.285257 29068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "e4525063167e4d0ba9fabf6f58769e67"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "8463930e2b626104fc5b7ede7f84f39d"
server_key_iv: "e6839c715c51e94c74f2e9efdf3b3f2f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.285853 29068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "e4525063167e4d0ba9fabf6f58769e67"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "8463930e2b626104fc5b7ede7f84f39d"
server_key_iv: "e6839c715c51e94c74f2e9efdf3b3f2f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.290130 29068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:29.293138 29083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.294802 29068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:29.294977 29068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "e4525063167e4d0ba9fabf6f58769e67"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "8463930e2b626104fc5b7ede7f84f39d"
server_key_iv: "e6839c715c51e94c74f2e9efdf3b3f2f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.295120 29068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:29.317271 29068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:29.320518 29068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:29.320732 29068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:29.328392 29068 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:34219
I20260619 06:53:29.328702 29135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:34219 every 8 connection(s)
I20260619 06:53:29.329504 29068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:29.333019 29136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:29.338662 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29068
I20260619 06:53:29.338840 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:29.339195 22980 external_mini_cluster.cc:1428] Setting key ae49b92401484b2ed67154f455aed9b7
I20260619 06:53:29.340195 29136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Bootstrap starting.
I20260619 06:53:29.343075 29136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:29.343998 29136 log.cc:824] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:29.346208 29136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: No bootstrap required, opened a new log
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:29.350978 29136 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } }
I20260619 06:53:29.351294 29136 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:29.351393 29136 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4525063167e4d0ba9fabf6f58769e67, State: Initialized, Role: FOLLOWER
I20260619 06:53:29.351913 29136 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } }
I20260619 06:53:29.352088 29136 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:29.352182 29136 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:29.352320 29136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:29.353411 29136 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } }
I20260619 06:53:29.353823 29136 leader_election.cc:302] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4525063167e4d0ba9fabf6f58769e67; no voters:
I20260619 06:53:29.354257 29136 leader_election.cc:288] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:29.354358 29141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:29.354564 29141 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 LEADER]: Becoming Leader. State: Replica: e4525063167e4d0ba9fabf6f58769e67, State: Running, Role: LEADER
I20260619 06:53:29.354943 29141 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } }
I20260619 06:53:29.355366 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:29.341039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.22.113.62:34219)
0619 06:53:29.341596 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:29.341614 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:29.341656 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:53:29.342466 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:29.342484 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:29.342881 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:29.343257 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:29.344158 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.345182 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.346025 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.346575 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.346758 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:29.349791 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:29.349796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:29.349810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:29.349838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:29.351922 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.352404 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.352412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.352418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.352557 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.352845 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.352849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.352851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.353303 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:29.353521 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:29.353782 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:53:29.354121 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":174,"threads_started":1}
I20260619 06:53:29.355861 29136 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:29.357043 29143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } }
I20260619 06:53:29.357218 29143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:29.357549 29142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4525063167e4d0ba9fabf6f58769e67. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } }
I20260619 06:53:29.357632 29142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:29.358227 29150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:29.361645 29150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:29.367708 29150 catalog_manager.cc:1347] Generated new cluster ID: a698d9a104c3492490b3cc3f5fe6081e
I20260619 06:53:29.367817 29150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:29.381528 29150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:29.382993 29150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:29.394150 29150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Generated new TSK 0
I20260619 06:53:29.395202 29150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
I20260619 06:53:29.426919 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:29.541453 29162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:29.541751 29162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:29.541796 29162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:29.547411 29162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:29.547488 29162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:29.547585 29162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:29.553225 29162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:29.554352 29162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:29.555449 29162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:29.565587 29168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:29.565666 29170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:29.565579 29167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:29.566094 29162 server_base.cc:1034] running on GCE node
I20260619 06:53:29.566542 29162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:29.567184 29162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:29.568434 29162 hybrid_clock.cc:648] HybridClock initialized: now 1781852009568383 us; error 64 us; skew 500 ppm
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:29.571864 29162 init.cc:377] Logged in from keytab as oryx/127.22.113.1@KRBTEST.COM (short username oryx)
I20260619 06:53:29.573017 29162 webserver.cc:460] Webserver started at http://127.22.113.1:39631/ using document root <none> and password file <none>
I20260619 06:53:29.573627 29162 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:29.573697 29162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:29.573930 29162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:29.576079 29162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "bc6f245104c74b49ae9ba3828a81b947"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "a75e584ccacae63bd99d7bc01233e943"
server_key_iv: "1fba5101118ba6ea6ed519576df7a63f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.576627 29162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6f245104c74b49ae9ba3828a81b947"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "a75e584ccacae63bd99d7bc01233e943"
server_key_iv: "1fba5101118ba6ea6ed519576df7a63f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.580518 29162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:29.583068 29177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.584632 29162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:29.584789 29162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "bc6f245104c74b49ae9ba3828a81b947"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "a75e584ccacae63bd99d7bc01233e943"
server_key_iv: "1fba5101118ba6ea6ed519576df7a63f"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.584908 29162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:29.622143 29162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:29.625540 29162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:29.625784 29162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:29.626447 29162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:29.627445 29162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:29.627565 29162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.627658 29162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:29.627722 29162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.638343 29162 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:39269
I20260619 06:53:29.638479 29290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:39269 every 8 connection(s)
I20260619 06:53:29.639432 29162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:29.644243 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29162
I20260619 06:53:29.644411 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:29.644793 22980 external_mini_cluster.cc:1428] Setting key 8d747266e0e0cc11f3b751ea3819c369
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:29.656350 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:29.641673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:34795 (local address 127.22.113.62:34219)
0619 06:53:29.641863 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:29.641867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:29.642771 (+ 904us) server_negotiation.cc:407] Connection header received
0619 06:53:29.644420 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:29.644424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:29.644490 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:29.644611 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:29.646054 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.646706 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.647996 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.648246 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.648361 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:29.651747 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:29.651750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:29.651752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:29.651776 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:29.653627 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.654274 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.654280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.654282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.654385 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.654786 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.654791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.654795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.655063 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:29.655184 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:29.655952 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 06:53:29.656126 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for oryx/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:53:29.657192 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:29.642052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:34795)
0619 06:53:29.642563 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:29.642613 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:29.644135 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:29.644814 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:29.644821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:29.645247 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:29.645910 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:29.645925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.646849 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.646853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:29.647865 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:29.647875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.648404 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.648409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:29.648586 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:29.649377 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:29.649403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:29.651526 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:29.653791 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:29.653796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:29.653811 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:29.654140 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:29.654498 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:29.654502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:29.654504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:29.654655 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:29.655211 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:29.655216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:29.655651 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 06:53:29.655853 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 06:53:29.656172 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260619 06:53:29.658778 29291 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:29.659060 29291 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:29.659862 29291 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:29.661793 29097 ts_manager.cc:194] Registered new tserver with Master: bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:29.663620 29097 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:34795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
I20260619 06:53:29.678303 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:29.808672 29296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:29.808971 29296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:29.809018 29296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:29.813351 29296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:29.813426 29296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:29.813494 29296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:29.819083 29296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:29.820432 29296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:29.821597 29296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:29.832756 29301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:29.832779 29304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:29.832715 29302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:29.832917 29296 server_base.cc:1034] running on GCE node
I20260619 06:53:29.833539 29296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:29.834287 29296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:29.835484 29296 hybrid_clock.cc:648] HybridClock initialized: now 1781852009835443 us; error 52 us; skew 500 ppm
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:29.838726 29296 init.cc:377] Logged in from keytab as oryx/127.22.113.2@KRBTEST.COM (short username oryx)
I20260619 06:53:29.839787 29296 webserver.cc:460] Webserver started at http://127.22.113.2:39847/ using document root <none> and password file <none>
I20260619 06:53:29.840467 29296 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:29.840564 29296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:29.840790 29296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:29.842947 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "375b00f6d236497baea694a3c907dea8"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "1233fcd961d34d7da4c38ccfd147b56d"
server_key_iv: "37cc007e1cdc39f9df5f5dcee7137cee"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.843565 29296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "375b00f6d236497baea694a3c907dea8"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "1233fcd961d34d7da4c38ccfd147b56d"
server_key_iv: "37cc007e1cdc39f9df5f5dcee7137cee"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.847635 29296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:29.850272 29311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.851792 29296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:29.851941 29296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "375b00f6d236497baea694a3c907dea8"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "1233fcd961d34d7da4c38ccfd147b56d"
server_key_iv: "37cc007e1cdc39f9df5f5dcee7137cee"
server_key_version: "encryptionkey@0"
I20260619 06:53:29.852070 29296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:29.869100 29296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:29.872431 29296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:29.872643 29296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:29.873287 29296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:29.874167 29296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:29.874245 29296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.874318 29296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:29.874367 29296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:29.884466 29296 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:36353
I20260619 06:53:29.884543 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:36353 every 8 connection(s)
I20260619 06:53:29.885461 29296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:29.886299 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29296
I20260619 06:53:29.886438 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:29.886796 22980 external_mini_cluster.cc:1428] Setting key 3819d6f34bf967578ee9a6e5fb6d9f47
Jun 19 06:53:29 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.3@KRBTEST.COM" created.
I20260619 06:53:29.903820 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:29.887953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:45157 (local address 127.22.113.62:34219)
0619 06:53:29.888160 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:29.888165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:29.889271 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 06:53:29.891641 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:29.891645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:29.891713 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:29.891804 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:29.893269 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.894212 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.895644 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.896102 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.896250 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:29.899406 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:29.899409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:29.899411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:29.899441 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:29.901239 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.901829 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.901837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.901839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.901932 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:29.902302 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:29.902305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:29.902307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:29.902571 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:29.902680 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:29.903521 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 06:53:29.903664 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:53:29.905165 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:29.888383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:45157)
0619 06:53:29.889040 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:29.889110 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:29.891344 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:29.892042 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:29.892051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:29.892486 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:29.893096 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:29.893111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.894414 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.894418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:29.895491 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:29.895505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:29.896278 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:29.896281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:29.896433 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:29.897092 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:29.897108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:29.899146 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:29.901371 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:29.901377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:29.901391 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:29.901685 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:29.902031 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:29.902035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:29.902037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:29.902192 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:29.902683 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:29.902693 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:29.903123 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 06:53:29.903389 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 06:53:29.903853 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":155,"threads_started":1}
I20260619 06:53:29.906885 29425 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:29.907196 29425 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:29.907953 29425 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:29.909382 29097 ts_manager.cc:194] Registered new tserver with Master: 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353)
I20260619 06:53:29.910149 29097 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:45157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
I20260619 06:53:29.922437 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:30.036844 29430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:30.037171 29430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:30.037253 29430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:30.041647 29430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:30.041762 29430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:30.041862 29430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:30.047214 29430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:30.048379 29430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:30.049417 29430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.059926 29435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.059962 29438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.059969 29436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.060102 29430 server_base.cc:1034] running on GCE node
I20260619 06:53:30.060626 29430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.061288 29430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.062479 29430 hybrid_clock.cc:648] HybridClock initialized: now 1781852010062456 us; error 37 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.065284 29430 init.cc:377] Logged in from keytab as oryx/127.22.113.3@KRBTEST.COM (short username oryx)
I20260619 06:53:30.066192 29430 webserver.cc:460] Webserver started at http://127.22.113.3:39811/ using document root <none> and password file <none>
I20260619 06:53:30.066771 29430 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.066864 29430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.067097 29430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:30.069181 29430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "e1f4bda6552f4989a145d30cf73f8085"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "b2d964484a461d7e5ab8cc0a6406e2cb"
server_key_iv: "1c526357ee9e0cc1d76a9d820ed59012"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.069754 29430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "e1f4bda6552f4989a145d30cf73f8085"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "b2d964484a461d7e5ab8cc0a6406e2cb"
server_key_iv: "1c526357ee9e0cc1d76a9d820ed59012"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.073689 29430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:30.076316 29445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.077623 29430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:30.077762 29430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "e1f4bda6552f4989a145d30cf73f8085"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "b2d964484a461d7e5ab8cc0a6406e2cb"
server_key_iv: "1c526357ee9e0cc1d76a9d820ed59012"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.077872 29430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.097494 29430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:30.100786 29430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:30.100983 29430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:30.101565 29430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:30.102392 29430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:30.102468 29430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.102528 29430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:30.102582 29430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.112721 29430 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:45993
I20260619 06:53:30.112788 29558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:45993 every 8 connection(s)
I20260619 06:53:30.113675 29430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:30.120244 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29430
I20260619 06:53:30.120374 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:30.120697 22980 external_mini_cluster.cc:1428] Setting key 98f34e62606c37547092e6204e2cc8e1
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:30.129343 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.115864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46513 (local address 127.22.113.62:34219)
0619 06:53:30.116046 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.116050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.116876 (+ 826us) server_negotiation.cc:407] Connection header received
0619 06:53:30.118438 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.118442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.118506 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.118620 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.120025 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.120748 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.122259 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.122524 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.122611 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.125521 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.125524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.125527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.125550 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.126999 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.127628 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.127634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.127636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.127716 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.128088 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.128092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.128094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.128295 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.128418 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.128983 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.129138 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:53:30.130347 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.116220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:46513)
0619 06:53:30.116697 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.116746 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.118181 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.118822 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.118836 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.119292 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.119861 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.119874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.120946 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.120951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.122123 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.122135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.122686 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.122690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.122890 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.123639 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.123658 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.125326 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.127123 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.127130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.127147 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.127523 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.127818 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.127822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.127824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.127986 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.128413 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.128421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.128717 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.128934 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.129341 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260619 06:53:30.131744 29559 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:30.132050 29559 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:30.132622 29559 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:30.133865 29097 ts_manager.cc:194] Registered new tserver with Master: e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993)
I20260619 06:53:30.134665 29097 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:46513
I20260619 06:53:30.135401 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.61.keytab.
Entry for principal oryx/127.22.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.61.keytab.
I20260619 06:53:30.163077 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:42915
--webserver_interface=127.22.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.61.keytab
--principal=oryx/127.22.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34219,127.22.113.61:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:30.273125 29564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:30.273428 29564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:30.273470 29564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:30.277566 29564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:30.277634 29564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:30.277655 29564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:30.277673 29564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:30.277742 29564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:30.283186 29564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34219,127.22.113.61:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.61.keytab
--principal=oryx/127.22.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
W20260619 06:53:30.283480 29564 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.113.62:34219,127.22.113.61:42915'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:53:30.284356 29564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:30.285362 29564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.294492 29570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.294458 29569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.294636 29564 server_base.cc:1034] running on GCE node
W20260619 06:53:30.294870 29572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.295629 29564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.296964 29564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.298130 29564 hybrid_clock.cc:648] HybridClock initialized: now 1781852010298098 us; error 56 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.301771 29564 init.cc:377] Logged in from keytab as oryx/127.22.113.61@KRBTEST.COM (short username oryx)
I20260619 06:53:30.302872 29564 webserver.cc:460] Webserver started at http://127.22.113.61:35859/ using document root <none> and password file <none>
I20260619 06:53:30.303579 29564 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.303650 29564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.303851 29564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:30.306183 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/instance:
uuid: "b01fd4ae0d8e45e085983ed02686defb"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "6c98bebce00f5b21f24d10cee1d622cd"
server_key_iv: "b73f1179a65450cf99d8c432d35a6906"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.306749 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal/instance:
uuid: "b01fd4ae0d8e45e085983ed02686defb"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "6c98bebce00f5b21f24d10cee1d622cd"
server_key_iv: "b73f1179a65450cf99d8c432d35a6906"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.311105 29564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:53:30.313901 29579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.315500 29564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:30.315611 29564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "b01fd4ae0d8e45e085983ed02686defb"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "6c98bebce00f5b21f24d10cee1d622cd"
server_key_iv: "b73f1179a65450cf99d8c432d35a6906"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.315707 29564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.341434 29564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:30.344892 29564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:30.345062 29564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:30.353426 29564 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.61:42915
I20260619 06:53:30.353462 29631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.61:42915 every 8 connection(s)
I20260619 06:53:30.354521 29564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
I20260619 06:53:30.357766 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:30.360770 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29564
I20260619 06:53:30.361917 29632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.61@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
I20260619 06:53:30.375042 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.363967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.22.113.62:34219)
0619 06:53:30.364124 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.364128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.364440 (+ 312us) server_negotiation.cc:407] Connection header received
0619 06:53:30.364779 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.364784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.364846 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.364961 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.366006 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.366574 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.367788 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.367997 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.368084 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.371015 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.371019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.371023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.371045 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.372612 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.373200 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.373204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.373206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.373291 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.373666 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.373669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.373671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.373899 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.374067 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.374715 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.374856 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:53:30.376101 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.363856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45706)
0619 06:53:30.364242 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.364295 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.364527 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.365031 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.365036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.365488 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.365832 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.365845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.366716 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.366719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.367648 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.367674 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.368157 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.368161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.368380 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.369139 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.369155 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.370803 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.372737 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.372747 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.372757 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.373073 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.373397 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.373403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.373405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.373551 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.374068 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.374078 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.374442 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.374627 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.374971 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":45,"thread_start_us":107,"threads_started":1}
I20260619 06:53:30.376099 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.362470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.22.113.61:42915)
0619 06:53:30.363014 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.363042 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.363085 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:53:30.364076 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.364104 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.364547 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.364808 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.365727 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.366789 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.367708 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.368179 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.368388 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.371641 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.371647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.371661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.371689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.373497 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.373990 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.373998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.374003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.374102 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.374402 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.374404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.374406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.374731 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.374944 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.375355 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.375496 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":369,"thread_start_us":160,"threads_started":1}
I20260619 06:53:30.377794 29632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.61@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
I20260619 06:53:30.386250 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.378399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:53960)
0619 06:53:30.378529 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.378544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.378619 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.378932 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.378934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.379067 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.379166 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.379171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.380300 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.380302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.380850 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.380857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.381361 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.381363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.381442 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.381874 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.381884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.383301 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.384993 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.384995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.384996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.385170 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.385407 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.385409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.385410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.385479 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.385822 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.385824 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.385896 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.385972 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.386078 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:53:30.386276 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.378613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.22.113.61:42915)
0619 06:53:30.378746 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.378749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.378761 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:30.378792 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.378794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.378827 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.378897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.379311 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.380200 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.380952 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.381261 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.381344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.383439 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.383442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.383444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.383464 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.384900 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.385256 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.385259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.385260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.385325 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.385580 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.385582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.385583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.385734 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.385814 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.386014 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.386134 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":58}
I20260619 06:53:30.389613 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Bootstrap starting.
I20260619 06:53:30.392309 29632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:30.393126 29632 log.cc:824] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:30.395449 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: No bootstrap required, opened a new log
I20260619 06:53:30.398715 29632 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } }
I20260619 06:53:30.398936 29632 raft_consensus.cc:374] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:30.398998 29632 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01fd4ae0d8e45e085983ed02686defb, State: Initialized, Role: FOLLOWER
I20260619 06:53:30.399519 29632 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } }
I20260619 06:53:30.400070 29638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } } }
I20260619 06:53:30.400264 29638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:30.400820 29632 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:53:30.405781 29649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:30.405889 29649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:30.408049 29564 master_runner.cc:428] Detected that this master b01fd4ae0d8e45e085983ed02686defb is joining an existing cluster
I20260619 06:53:30.408124 29564 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.113.61:42915
I20260619 06:53:30.419148 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.410231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.22.113.61:42915)
0619 06:53:30.410373 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.410380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.411415 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 06:53:30.411507 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.411510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.411555 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.411633 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.412067 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.412569 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.413936 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.414152 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.414204 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.415808 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.415812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.415814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.415834 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.417416 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.417863 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.417869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.417874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.417958 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.418239 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.418242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.418244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.418439 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.418532 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.418848 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.419005 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":1792}
I20260619 06:53:30.419162 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.410559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:53976)
0619 06:53:30.411295 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.411311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.411426 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.411716 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.411717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.411879 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.411960 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.411965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.412703 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.412705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.413815 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.413826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.414317 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.414319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.414412 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.415161 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.415178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.415672 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.417543 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.417546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.417549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.417770 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.418049 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.418062 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.418064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.418146 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.418584 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.418586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.418683 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.418806 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.418954 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":100,"spinlock_wait_cycles":1280,"thread_start_us":130,"threads_started":1}
I20260619 06:53:30.419940 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.410232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.22.113.62:34219)
0619 06:53:30.410472 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.410476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.410803 (+ 327us) server_negotiation.cc:407] Connection header received
0619 06:53:30.410887 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.410891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.410950 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.411044 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.411650 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.412531 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.413947 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.414303 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.414425 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.415829 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.415834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.415837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.415867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.417777 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.418338 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.418342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.418344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.418450 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.418785 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.418790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.418792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.419084 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.419181 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.419642 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.419799 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260619 06:53:30.419970 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.410171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45708)
0619 06:53:30.410498 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.410514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.410644 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.411075 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.411078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.411320 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.411509 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.411516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.412657 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.412662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.413817 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.413826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.414437 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.414440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.414553 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.415049 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.415074 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.415672 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.417917 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.417921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.417924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.418192 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.418588 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.418590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.418592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.418679 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.419226 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.419229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.419375 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.419518 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.419762 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":79,"spinlock_wait_cycles":2304,"thread_start_us":113,"threads_started":1}
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.62@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
I20260619 06:53:30.436370 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.425074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:53984)
0619 06:53:30.425494 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.425550 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.425702 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.426036 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.426042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.426529 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.426669 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.426677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.427463 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.427466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.428509 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.428520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.429068 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.429073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.429213 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.429896 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.429912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.431916 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.434027 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.434037 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.434045 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.434402 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.434730 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.434733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.434736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.434835 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.435434 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.435444 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.435646 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.435836 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.436101 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20260619 06:53:30.437115 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.425113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.22.113.61:42915)
0619 06:53:30.425276 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.425281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.425652 (+ 371us) server_negotiation.cc:407] Connection header received
0619 06:53:30.425853 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.425856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.425898 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.425978 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.426799 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.427357 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.428643 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.428939 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.429059 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.432076 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.432081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.432085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.432112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.433897 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.434519 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.434526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.434529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.434616 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.434945 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.434951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.434955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.435214 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.435352 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.436831 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.436951 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.22.113.60@KRBTEST.COM; defaulting to no policy
I20260619 06:53:30.438874 29100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.113.61:42915: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb"
member_type: NON_VOTER
last_known_addr {
host: "127.22.113.61"
port: 42915
}
attrs {
promote: true
}
}
dest_uuid: "e4525063167e4d0ba9fabf6f58769e67"
cas_config_opid_index: -1
I20260619 06:53:30.439803 29100 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.60@KRBTEST.COM" created.
I20260619 06:53:30.442659 29660 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b01fd4ae0d8e45e085983ed02686defb (127.22.113.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } }
I20260619 06:53:30.443183 29606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:30.444489 29606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:53:30.444617 29660 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.113.61:42915
I20260619 06:53:30.444854 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } }
I20260619 06:53:30.444950 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:30.445124 29564 master.cc:537] Master@127.22.113.61:42915 shutting down...
I20260619 06:53:30.445273 29661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:30.447362 29085 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 06:53:30.450021 29564 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:30.450350 29564 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:30.450467 29564 tablet_replica.cc:330] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: stopping tablet replica
I20260619 06:53:30.456136 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.455646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:53998)
0619 06:53:30.455901 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.456030 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 06:53:30.456295 29085 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 -> Peer b01fd4ae0d8e45e085983ed02686defb (127.22.113.61:42915): Couldn't send request to peer b01fd4ae0d8e45e085983ed02686defb. Status: Network error: Client connection negotiation failed: client connection to 127.22.113.61:42915: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.60.keytab.
Entry for principal oryx/127.22.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.60.keytab.
I20260619 06:53:30.462877 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:38709
--webserver_interface=127.22.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.60.keytab
--principal=oryx/127.22.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:53:30.466413 29564 master.cc:552] Master@127.22.113.61:42915 shutdown complete.
I20260619 06:53:30.466492 29564 master_runner.cc:305] Clearing existing system tablet
I20260619 06:53:30.466777 29564 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.466845 29564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.468133 29564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 06:53:30.469018 29667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.469230 29564 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.469384 29564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:30.469422 29564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "b01fd4ae0d8e45e085983ed02686defb"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "6c98bebce00f5b21f24d10cee1d622cd"
server_key_iv: "b73f1179a65450cf99d8c432d35a6906"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.471220 29564 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:30.473209 29564 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:53:30.473317 29564 log.cc:1196] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:53:30.473567 29564 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Deleting consensus metadata
I20260619 06:53:30.473748 29564 master_runner.cc:315] Copying system tablet from 127.22.113.62:34219
I20260619 06:53:30.474900 29564 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: tablet copy: Beginning tablet copy session from remote peer at address 127.22.113.62:34219
I20260619 06:53:30.485106 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.475567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.22.113.62:34219)
0619 06:53:30.475692 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.475696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.477965 (+ 2269us) server_negotiation.cc:407] Connection header received
0619 06:53:30.478127 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.478130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.478185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.478270 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.478758 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.479394 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.480215 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.480447 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.480553 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.481801 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.481805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.481807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.481831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.483501 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.483926 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.483930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.483932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.484020 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.484291 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.484295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.484297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.484471 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.484577 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.484841 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.484958 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:53:30.485117 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.475518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45724)
0619 06:53:30.477851 (+ 2333us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.477864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.478015 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.478285 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.478288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.478538 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.478639 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.478645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.479506 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.479513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.480098 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.480106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.480560 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.480563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.480656 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.481203 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.481220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.481668 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.483623 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.483626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.483629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.483827 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.484111 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.484115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.484117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.484200 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.484566 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.484569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.484672 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.484776 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.484915 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"thread_start_us":86,"threads_started":1}
I20260619 06:53:30.485977 29120 tablet_copy_service.cc:140] P e4525063167e4d0ba9fabf6f58769e67: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b01fd4ae0d8e45e085983ed02686defb ({username='oryx', principal='oryx/127.22.113.61@KRBTEST.COM'} at 127.0.0.1:45724)
I20260619 06:53:30.486155 29120 tablet_copy_service.cc:161] P e4525063167e4d0ba9fabf6f58769e67: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b01fd4ae0d8e45e085983ed02686defb at {username='oryx', principal='oryx/127.22.113.61@KRBTEST.COM'} at 127.0.0.1:45724: session id = b01fd4ae0d8e45e085983ed02686defb-00000000000000000000000000000000
I20260619 06:53:30.487613 29120 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:53:30.489188 29564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:53:30.491220 29564 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: tablet copy: Starting download of 0 data blocks...
I20260619 06:53:30.491473 29564 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: tablet copy: Starting download of 1 WAL segments...
I20260619 06:53:30.493101 29564 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:53:30.494649 29120 tablet_copy_service.cc:342] P e4525063167e4d0ba9fabf6f58769e67: Request end of tablet copy session b01fd4ae0d8e45e085983ed02686defb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.113.61@KRBTEST.COM'} at 127.0.0.1:45724
I20260619 06:53:30.494763 29120 tablet_copy_service.cc:433] P e4525063167e4d0ba9fabf6f58769e67: ending tablet copy session b01fd4ae0d8e45e085983ed02686defb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b01fd4ae0d8e45e085983ed02686defb
I20260619 06:53:30.502363 29564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.504056 29674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.504115 29677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.504384 29675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.504657 29564 server_base.cc:1034] running on GCE node
I20260619 06:53:30.504823 29564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.505144 29564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.506332 29564 hybrid_clock.cc:648] HybridClock initialized: now 1781852010506344 us; error 51 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.507931 29564 init.cc:377] Logged in from keytab as oryx/127.22.113.61@KRBTEST.COM (short username oryx)
I20260619 06:53:30.508435 29564 webserver.cc:460] Webserver started at http://127.22.113.61:33359/ using document root <none> and password file <none>
I20260619 06:53:30.508618 29564 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.508673 29564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.509899 29564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:30.510820 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.511147 29564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:30.511240 29564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "b01fd4ae0d8e45e085983ed02686defb"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "6c98bebce00f5b21f24d10cee1d622cd"
server_key_iv: "b73f1179a65450cf99d8c432d35a6906"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.511368 29564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.524829 29564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:30.526810 29564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:30.531782 29564 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.61:42915
I20260619 06:53:30.531790 29736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.61:42915 every 8 connection(s)
I20260619 06:53:30.532343 29564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
I20260619 06:53:30.533875 29737 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:53:30.534723 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Bootstrap starting.
I20260619 06:53:30.542786 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:53:30.543072 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: Bootstrap complete.
I20260619 06:53:30.543643 29737 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } }
I20260619 06:53:30.543773 29737 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b01fd4ae0d8e45e085983ed02686defb, State: Initialized, Role: LEARNER
I20260619 06:53:30.543887 29737 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } }
I20260619 06:53:30.544401 29737 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:30.544545 29740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } }
I20260619 06:53:30.544709 29740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: LEARNER
I20260619 06:53:30.547345 29751 catalog_manager.cc:1259] Loaded cluster ID: a698d9a104c3492490b3cc3f5fe6081e
I20260619 06:53:30.547863 29751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: loading cluster ID for follower catalog manager: success
I20260619 06:53:30.550215 29751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: acquiring CA information for follower catalog manager: success
I20260619 06:53:30.551717 29751 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:53:30.630705 29665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:30.631006 29665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:30.631085 29665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:30.635321 29665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:30.635394 29665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:30.635416 29665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:30.635433 29665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:30.635490 29665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:30.641937 29665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.60.keytab
--principal=oryx/127.22.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:38709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:30.643297 29665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:30.644403 29665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.653981 29759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.654027 29756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.653996 29757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.654443 29665 server_base.cc:1034] running on GCE node
I20260619 06:53:30.655287 29665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.656839 29665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.658057 29665 hybrid_clock.cc:648] HybridClock initialized: now 1781852010658034 us; error 63 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.661186 29665 init.cc:377] Logged in from keytab as oryx/127.22.113.60@KRBTEST.COM (short username oryx)
I20260619 06:53:30.662236 29665 webserver.cc:460] Webserver started at http://127.22.113.60:43099/ using document root <none> and password file <none>
I20260619 06:53:30.662878 29665 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.662954 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.663224 29665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:30.665660 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/instance:
uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "05554b61eff13f6729fddf0c1045c48b"
server_key_iv: "bda9fe84943e570bbe3c435a4711ca53"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.666290 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal/instance:
uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "05554b61eff13f6729fddf0c1045c48b"
server_key_iv: "bda9fe84943e570bbe3c435a4711ca53"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.667196 29291 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
I20260619 06:53:30.670890 29665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:53:30.673681 29766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.675158 29665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:30.675364 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "05554b61eff13f6729fddf0c1045c48b"
server_key_iv: "bda9fe84943e570bbe3c435a4711ca53"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.675472 29665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.696470 29665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:30.700042 29665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:30.700208 29665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:30.708000 29665 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.60:38709
I20260619 06:53:30.708381 29818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.60:38709 every 8 connection(s)
I20260619 06:53:30.709172 29665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
I20260619 06:53:30.711570 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29665
I20260619 06:53:30.712805 29819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:30.718384 29819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.60@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:30.728327 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.713275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.22.113.60:38709)
0619 06:53:30.713747 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.713769 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.713825 (+ 56us) server_negotiation.cc:407] Connection header received
0619 06:53:30.714825 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.714849 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.715392 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.715841 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.716614 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.718171 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.719152 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.719878 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.720042 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.722981 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.722986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.722995 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.723021 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.724831 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.725349 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.725353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.725357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.725477 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.725790 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.725793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.725794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.726209 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.726429 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.726850 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.727186 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260619 06:53:30.730846 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.720773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.22.113.62:34219)
0619 06:53:30.720956 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.720962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.721351 (+ 389us) server_negotiation.cc:407] Connection header received
0619 06:53:30.721557 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.721560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.721612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.721663 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.722655 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.723367 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.724372 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.724592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.724701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.727189 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.727193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.727196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.727232 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.728934 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.729440 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.729445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.729447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.729535 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.729857 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.729862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.729864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.730062 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.730174 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.730515 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.730646 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:53:30.730880 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.720662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45734)
0619 06:53:30.721191 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.721251 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.721449 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.721844 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.721850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.722324 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.722486 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.722493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.723556 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.723560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.724244 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.724254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.724704 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.724708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.724832 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.725369 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.725383 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.726985 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.729065 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.729076 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.729083 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.729328 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.729640 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.729643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.729645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.729730 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.730172 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.730179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.730302 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.730451 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.730675 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":181,"threads_started":1}
I20260619 06:53:30.732167 29819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.60@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
I20260619 06:53:30.741531 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.732795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:54000)
0619 06:53:30.732971 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.732989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.733073 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.733648 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.733650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.733792 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.733897 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.733902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.734904 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.734906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.735558 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.735566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.736024 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.736026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.736108 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.736592 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.736604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.738080 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.740113 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.740116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.740118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.740331 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.740621 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.740623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.740624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.740699 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.741074 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.741076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.741177 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.741269 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.741389 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 06:53:30.741621 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.733020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.22.113.61:42915)
0619 06:53:30.733378 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.733382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.733397 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:53:30.733438 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.733440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.733521 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.733614 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.734025 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.734788 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.735672 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.735914 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.736009 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.738255 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.738259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.738270 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.738298 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.739991 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.740429 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.740432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.740433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.740506 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.740784 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.740786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.740787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.740978 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.741056 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.741326 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.741440 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":153,"threads_started":1}
I20260619 06:53:30.742450 29819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.60@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
I20260619 06:53:30.752108 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.743050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41784)
0619 06:53:30.743184 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.743201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.743364 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.743635 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.743637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.743818 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.743882 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.743887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.745363 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.745366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.746005 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.746015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.746586 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.746589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.746656 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.747136 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.747147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.748627 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.750655 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.750658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.750659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.750877 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.751148 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.751150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.751151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.751222 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.751677 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.751679 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.751766 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.751850 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.751966 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:53:30.752220 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.743190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.22.113.60:38709)
0619 06:53:30.743352 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.743357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.743370 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:53:30.743453 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.743456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.743538 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.743625 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.743989 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.745156 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.746131 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.746414 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.746568 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.748786 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.748789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.748792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.748814 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.750499 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.750980 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.750982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.750984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.751060 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.751379 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.751381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.751382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.751585 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.751675 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.751933 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.752040 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 06:53:30.756003 29819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Bootstrap starting.
I20260619 06:53:30.759384 29819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:30.760330 29819 log.cc:824] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:30.762816 29819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: No bootstrap required, opened a new log
I20260619 06:53:30.766933 29819 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } }
I20260619 06:53:30.767236 29819 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:30.767361 29819 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb8e18be3f274c4ea3ed6bb9e7e7adb9, State: Initialized, Role: FOLLOWER
I20260619 06:53:30.767863 29819 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } }
I20260619 06:53:30.768399 29825 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } } }
I20260619 06:53:30.768540 29825 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:30.769387 29819 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:30.772106 29825 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:30.772531 29825 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } }
I20260619 06:53:30.773653 29825 leader_election.cc:288] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219), b01fd4ae0d8e45e085983ed02686defb (127.22.113.61:42915)
I20260619 06:53:30.774318 29711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01fd4ae0d8e45e085983ed02686defb" is_pre_election: true
I20260619 06:53:30.774526 29711 raft_consensus.cc:1799] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Handling vote request from an unknown peer fb8e18be3f274c4ea3ed6bb9e7e7adb9
I20260619 06:53:30.774700 29711 raft_consensus.cc:2396] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate fb8e18be3f274c4ea3ed6bb9e7e7adb9 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 06:53:30.774852 29110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4525063167e4d0ba9fabf6f58769e67" is_pre_election: true
W20260619 06:53:30.775091 29836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:30.775173 29836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:30.775725 29770 leader_election.cc:302] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb8e18be3f274c4ea3ed6bb9e7e7adb9; no voters: b01fd4ae0d8e45e085983ed02686defb, e4525063167e4d0ba9fabf6f58769e67
I20260619 06:53:30.776052 29825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:30.778080 29825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:53:30.780606 29665 master_runner.cc:428] Detected that this master fb8e18be3f274c4ea3ed6bb9e7e7adb9 is joining an existing cluster
I20260619 06:53:30.780683 29665 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.113.60:38709
I20260619 06:53:30.784418 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29162
I20260619 06:53:30.792762 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:39269
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:34219
--webserver_port=39631
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:53:30.794826 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.783588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41788)
0619 06:53:30.785041 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.785059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.785189 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.785523 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.785528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.785823 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.785984 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.785994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.787161 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.787167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.788349 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.788362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.789234 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.789240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.789756 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.790414 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.790434 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.790984 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.793256 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.793259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.793262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.793508 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.793769 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.793772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.793774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.793844 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.794270 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.794273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.794344 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.794449 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.794602 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":133,"threads_started":1}
I20260619 06:53:30.795042 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.783933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.22.113.60:38709)
0619 06:53:30.784029 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.784033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.785125 (+ 1092us) server_negotiation.cc:407] Connection header received
0619 06:53:30.785280 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.785286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.785332 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.785381 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.786104 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.786998 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.788484 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.788972 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.789097 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.791162 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.791166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.791175 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.791202 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.793101 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.793605 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.793609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.793611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.793680 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.793937 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.793940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.793942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.794148 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.794213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.794579 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.794736 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":37}
I20260619 06:53:30.804616 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.783190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.22.113.62:34219)
0619 06:53:30.783338 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.783345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.795134 (+ 11789us) server_negotiation.cc:407] Connection header received
0619 06:53:30.795379 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.795386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.795451 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.795545 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.796067 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.796872 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.798074 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.798510 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.798639 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.800487 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.800493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.800499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.800531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.802398 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.802945 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.802951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.802955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.803062 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.803445 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.803451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.803454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.803714 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.803802 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.804244 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.804381 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:53:30.805262 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.784213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45736)
0619 06:53:30.795047 (+ 10834us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.795063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.795209 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.795586 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.795590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.795789 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.795915 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.795924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.797000 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.797003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.797931 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.797941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.798810 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.798817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.798963 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.799676 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.799696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.800310 (+ 614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.802535 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.802539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.802541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.802814 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.803193 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.803199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.803201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.803321 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.803886 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.803892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.804009 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.804807 (+ 798us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.805006 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10770,"thread_start_us":94,"threads_started":1}
I20260619 06:53:30.813110 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.783791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.22.113.61:42915)
0619 06:53:30.783927 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.783934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.784029 (+ 95us) server_negotiation.cc:407] Connection header received
0619 06:53:30.784224 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.784227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.784283 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.784351 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.785041 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.785836 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.788252 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.788612 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.788690 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.808407 (+ 19717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.808412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.808416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.808603 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.810598 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.811725 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.811729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.811731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.811814 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.812118 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.812121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.812123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.812366 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.812449 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.812764 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.812925 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:53:30.813953 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.783295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:54012)
0619 06:53:30.783942 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.783959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.784109 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.784522 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.784525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.784772 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.784908 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.784916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.785984 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.785987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.788103 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.788114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.788890 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.788894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.789048 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.789712 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.789727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.807897 (+ 18170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.811347 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.811350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.811353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.811607 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.811935 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.811938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.811940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.812024 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.812503 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.812506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.812594 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.813587 (+ 993us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.813752 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":94,"thread_start_us":141,"threads_started":1}
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852009, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.62@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
I20260619 06:53:30.820408 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.810067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41794)
0619 06:53:30.810299 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.810313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.810383 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.811213 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.811216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.811422 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.811523 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.811529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.812712 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.812716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.813645 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.813659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.814362 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.814366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.814456 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.815047 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.815064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.816706 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.818795 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.818800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.818802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.819036 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.819363 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.819366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.819368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.819454 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.819889 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.819892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.819984 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.820091 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.820232 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 06:53:30.820932 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.810773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.22.113.60:38709)
0619 06:53:30.810926 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.810931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.810944 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:53:30.811025 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.811028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.811084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.811173 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.811660 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.812534 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.813798 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.814160 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.814255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.816854 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.816859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.816862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.816887 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.818614 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.819156 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.819160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.819163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.819239 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.819551 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.819553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.819555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.819762 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.819848 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.820678 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.820789 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:53:30.822256 29100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.113.60:38709: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
member_type: NON_VOTER
last_known_addr {
host: "127.22.113.60"
port: 38709
}
attrs {
promote: true
}
}
dest_uuid: "e4525063167e4d0ba9fabf6f58769e67"
cas_config_opid_index: 5
I20260619 06:53:30.823094 29100 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } }
I20260619 06:53:30.824401 29661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.826155 29661 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.113.60:38709
I20260619 06:53:30.826341 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.826449 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:30.827402 29665 master.cc:537] Master@127.22.113.60:38709 shutting down...
I20260619 06:53:30.827766 29793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:53:30.828888 29660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:30.830612 29086 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 -> Peer fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60:38709): Couldn't send request to peer fb8e18be3f274c4ea3ed6bb9e7e7adb9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:53:30.831482 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.824539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:54028)
0619 06:53:30.824876 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.824891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.824966 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.827191 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.827195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.827213 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:30.827343 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.827349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.828462 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.828465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.829872 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.829883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.831055 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.831058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.831144 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.831185 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.831280 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.831349 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260619 06:53:30.831930 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.826661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.22.113.61:42915)
0619 06:53:30.826825 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.826831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.826845 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:53:30.826895 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.826898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.826958 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.827062 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:30.827471 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.828253 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.830013 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.830647 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.830847 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.830983 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.831744 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.831819 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:30.832773 29711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:53:30.833479 29660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:53:30.836078 29711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.838250 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.838470 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: LEARNER
I20260619 06:53:30.838567 29740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4525063167e4d0ba9fabf6f58769e67. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: NON_VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: true } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.838704 29740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: LEARNER
I20260619 06:53:30.840050 29665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:30.840345 29665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:30.840433 29665 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: stopping tablet replica
I20260619 06:53:30.845340 29661 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: attempting to promote NON_VOTER b01fd4ae0d8e45e085983ed02686defb to VOTER
I20260619 06:53:30.845924 29661 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } }
I20260619 06:53:30.846393 29665 master.cc:552] Master@127.22.113.60:38709 shutdown complete.
I20260619 06:53:30.846444 29665 master_runner.cc:305] Clearing existing system tablet
I20260619 06:53:30.846714 29665 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.846762 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.847790 29711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 LEARNER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:53:30.847924 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.847647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41802)
0619 06:53:30.847794 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.847848 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:53:30.848436 29660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:53:30.849262 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.848999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41818)
0619 06:53:30.849147 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.849191 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 06:53:30.849514 29086 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 -> Peer fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60:38709): Couldn't send request to peer fb8e18be3f274c4ea3ed6bb9e7e7adb9. Status: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38709: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:53:30.850314 29848 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b01fd4ae0d8e45e085983ed02686defb (127.22.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.850912 29711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b01fd4ae0d8e45e085983ed02686defb (127.22.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.852378 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.852497 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:30.852681 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.852754 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:30.852933 29740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.853055 29740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:30.864866 29665 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.002s sys 0.000s
I20260619 06:53:30.866119 29850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.866549 29665 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.866695 29665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:30.866734 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "05554b61eff13f6729fddf0c1045c48b"
server_key_iv: "bda9fe84943e570bbe3c435a4711ca53"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.869450 29665 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:30.872206 29665 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:53:30.872301 29665 log.cc:1196] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:53:30.872566 29665 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Deleting consensus metadata
I20260619 06:53:30.872748 29665 master_runner.cc:315] Copying system tablet from 127.22.113.62:34219
I20260619 06:53:30.874126 29665 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.113.62:34219
I20260619 06:53:30.885249 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.874856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.22.113.62:34219)
0619 06:53:30.875008 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:30.875014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:30.875643 (+ 629us) server_negotiation.cc:407] Connection header received
0619 06:53:30.875784 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:30.875788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:30.875850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:30.875963 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:30.876544 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.877445 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.878693 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.879067 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.879205 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.880838 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:30.880842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:30.880845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:30.880874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:30.883045 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.883574 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.883578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.883580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.883677 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:30.884036 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:30.884039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:30.884041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:30.884397 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:30.884512 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:30.884930 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 06:53:30.885070 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:53:30.886154 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:30.875140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:45750)
0619 06:53:30.875488 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:30.875508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:30.875676 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:30.875992 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:30.875995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:30.876264 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:30.876406 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.876414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.877579 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.877582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.878555 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:30.878565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:30.879217 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:30.879221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:30.879338 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:30.880023 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:30.880042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:30.880630 (+ 588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:30.883173 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.883176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.883179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.883459 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.883787 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:30.883790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:30.883792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:30.883910 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:30.884583 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:30.884587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:30.884723 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 06:53:30.885745 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0619 06:53:30.885947 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":165,"threads_started":1}
I20260619 06:53:30.886842 29120 tablet_copy_service.cc:140] P e4525063167e4d0ba9fabf6f58769e67: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb8e18be3f274c4ea3ed6bb9e7e7adb9 ({username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:45750)
I20260619 06:53:30.886956 29120 tablet_copy_service.cc:161] P e4525063167e4d0ba9fabf6f58769e67: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb8e18be3f274c4ea3ed6bb9e7e7adb9 at {username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:45750: session id = fb8e18be3f274c4ea3ed6bb9e7e7adb9-00000000000000000000000000000000
I20260619 06:53:30.888005 29120 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:53:30.889559 29665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:53:30.892391 29665 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: tablet copy: Starting download of 0 data blocks...
I20260619 06:53:30.892623 29665 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: tablet copy: Starting download of 1 WAL segments...
I20260619 06:53:30.894565 29665 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:53:30.897519 29120 tablet_copy_service.cc:342] P e4525063167e4d0ba9fabf6f58769e67: Request end of tablet copy session fb8e18be3f274c4ea3ed6bb9e7e7adb9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:45750
I20260619 06:53:30.897637 29120 tablet_copy_service.cc:433] P e4525063167e4d0ba9fabf6f58769e67: ending tablet copy session fb8e18be3f274c4ea3ed6bb9e7e7adb9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb8e18be3f274c4ea3ed6bb9e7e7adb9
I20260619 06:53:30.906762 29665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.909006 29857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.912325 29858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.912765 29425 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
W20260619 06:53:30.912958 29860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.913607 29665 server_base.cc:1034] running on GCE node
I20260619 06:53:30.914482 29665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.914888 29665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.916066 29665 hybrid_clock.cc:648] HybridClock initialized: now 1781852010916050 us; error 34 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.918040 29665 init.cc:377] Logged in from keytab as oryx/127.22.113.60@KRBTEST.COM (short username oryx)
I20260619 06:53:30.918576 29665 webserver.cc:460] Webserver started at http://127.22.113.60:36739/ using document root <none> and password file <none>
I20260619 06:53:30.918793 29665 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.918846 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.920898 29665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:53:30.922119 29867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:30.922551 29665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:30.922640 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "05554b61eff13f6729fddf0c1045c48b"
server_key_iv: "bda9fe84943e570bbe3c435a4711ca53"
server_key_version: "encryptionkey@0"
I20260619 06:53:30.922748 29665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:30.936362 29665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:30.938827 29665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:30.944797 29665 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.60:38709
I20260619 06:53:30.944962 29919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.60:38709 every 8 connection(s)
I20260619 06:53:30.945436 29665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
I20260619 06:53:30.947541 29920 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:53:30.948844 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Bootstrap starting.
W20260619 06:53:30.958794 29845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:30.959192 29845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:30.959298 29845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:53:30.961897 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:53:30.962375 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Bootstrap complete.
I20260619 06:53:30.963049 29920 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } }
I20260619 06:53:30.963351 29920 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb8e18be3f274c4ea3ed6bb9e7e7adb9, State: Initialized, Role: LEARNER
I20260619 06:53:30.963577 29920 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } }
I20260619 06:53:30.964084 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:30.964268 29920 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:30.964284 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: LEARNER
W20260619 06:53:30.966039 29845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:30.966122 29845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:30.966244 29845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:30.968591 29937 catalog_manager.cc:1259] Loaded cluster ID: a698d9a104c3492490b3cc3f5fe6081e
I20260619 06:53:30.968657 29937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: loading cluster ID for follower catalog manager: success
I20260619 06:53:30.970733 29937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: acquiring CA information for follower catalog manager: success
I20260619 06:53:30.972280 29937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:53:30.973047 29845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:39269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=39631
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:30.974272 29845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:30.975435 29845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:30.986538 29940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.986588 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:30.986588 29939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:30.986999 29845 server_base.cc:1034] running on GCE node
I20260619 06:53:30.987514 29845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:30.988204 29845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:30.989393 29845 hybrid_clock.cc:648] HybridClock initialized: now 1781852010989372 us; error 43 us; skew 500 ppm
Jun 19 06:53:30 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:30.992623 29845 init.cc:377] Logged in from keytab as oryx/127.22.113.1@KRBTEST.COM (short username oryx)
I20260619 06:53:30.993623 29845 webserver.cc:460] Webserver started at http://127.22.113.1:39631/ using document root <none> and password file <none>
I20260619 06:53:30.994221 29845 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:30.994335 29845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:30.998642 29845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:31.001181 29949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.002774 29845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:31.002955 29845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "bc6f245104c74b49ae9ba3828a81b947"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "a75e584ccacae63bd99d7bc01233e943"
server_key_iv: "1fba5101118ba6ea6ed519576df7a63f"
server_key_version: "encryptionkey@0"
I20260619 06:53:31.003604 29845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:31.017198 29845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:31.020356 29845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:31.020567 29845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:31.021163 29845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:31.022037 29845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:31.022111 29845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.022183 29845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:31.022239 29845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.032843 29845 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:39269
I20260619 06:53:31.032862 30062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:39269 every 8 connection(s)
I20260619 06:53:31.033882 29845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:31.042915 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 29845
I20260619 06:53:31.043150 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29296
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852010, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:31.052397 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.037517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:46053 (local address 127.22.113.62:34219)
0619 06:53:31.037668 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.037672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.037689 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:31.038721 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.038724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.038816 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.038908 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.040496 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.041071 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.043933 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.044253 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.044375 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.048089 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.048093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.048096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.048123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.049907 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050496 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.050583 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050922 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.051159 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.051270 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.052097 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.052227 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:53:31.052397 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.036663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:60971 (local address 127.22.113.61:42915)
0619 06:53:31.036843 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.036848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.037630 (+ 782us) server_negotiation.cc:407] Connection header received
0619 06:53:31.038572 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.038579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.038653 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.038772 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.040493 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.041100 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.042531 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.042908 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.043018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.047724 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.047729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.047732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.047759 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.049514 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050408 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.050503 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050925 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.051169 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.051273 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.052097 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.052227 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:53:31.052578 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.036663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:33625 (local address 127.22.113.60:38709)
0619 06:53:31.038308 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.038312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.038607 (+ 295us) server_negotiation.cc:407] Connection header received
0619 06:53:31.038827 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.038831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.038896 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.038984 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.041299 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.042003 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.043920 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.044358 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.044461 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.047725 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.047732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.047738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.047773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.049842 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050447 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.050551 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050934 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.050940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.050945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.051215 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.051331 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.052107 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.052280 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":115,"threads_started":1}
I20260619 06:53:31.054453 30068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.036417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.22.113.1:33625)
0619 06:53:31.038494 (+ 2077us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.038513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.038683 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.039090 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.039094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.039739 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.040335 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.040346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.042138 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.042142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.043754 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.043768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.044665 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.044671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.044813 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.045399 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.045419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.047494 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.049968 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.049972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.049974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050329 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.050658 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.050666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050828 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.051391 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.051398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.051796 (+ 398us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.052832 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.053264 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"mutex_wait_us":1206,"spinlock_wait_cycles":89088,"thread_start_us":120,"threads_started":1}
I20260619 06:53:31.054445 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.036416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:46053)
0619 06:53:31.037190 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.037266 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.038292 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.039018 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.039028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.039681 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.040319 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.040337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.042131 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.042134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.043806 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.043819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.045142 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.045149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.045310 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.045798 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.045816 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.047919 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.050031 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.050035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050388 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.050696 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.050700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050813 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.051336 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.051343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.051793 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.052561 (+ 768us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.053124 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":931,"spinlock_wait_cycles":5248,"thread_start_us":138,"threads_started":1}
I20260619 06:53:31.054447 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.036417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.22.113.1:60971)
0619 06:53:31.037542 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.037555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.038292 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.039018 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.039028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.039739 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.040337 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.040346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.041263 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.041266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.042382 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.042393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.043193 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.043197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.043507 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.044271 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.044295 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.047493 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.049657 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.049665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.049683 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050034 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.050615 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.050621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.050624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.050808 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.051336 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.051343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.051794 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.052601 (+ 807us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.053059 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":282,"spinlock_wait_cycles":33920,"thread_start_us":132,"threads_started":1}
I20260619 06:53:31.055969 30063 heartbeater.cc:347] Connected to a master server at 127.22.113.60:38709
I20260619 06:53:31.056032 30064 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:31.056219 30064 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.056247 30063 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.056810 30064 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:31.056808 30063 heartbeater.cc:511] Master 127.22.113.60:38709 requested a full tablet report, sending...
I20260619 06:53:31.057250 30065 heartbeater.cc:347] Connected to a master server at 127.22.113.61:42915
I20260619 06:53:31.057359 30065 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.057540 30065 heartbeater.cc:511] Master 127.22.113.61:42915 requested a full tablet report, sending...
I20260619 06:53:31.058492 29100 ts_manager.cc:194] Re-registered known tserver with Master: bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:31.058624 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:36353
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:34219
--webserver_port=39847
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:53:31.058630 29883 ts_manager.cc:194] Registered new tserver with Master: bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:31.058630 29701 ts_manager.cc:194] Registered new tserver with Master: bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:31.059566 29100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:46053
I20260619 06:53:31.137398 29559 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
W20260619 06:53:31.188364 30071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:31.188647 30071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:31.188689 30071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:31.192818 30071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:31.192889 30071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:31.192950 30071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:31.198467 30071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:36353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=39847
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:31.199875 30071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:31.201064 30071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:31.212249 30077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:31.212212 30079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:31.212229 30076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:31.212683 30071 server_base.cc:1034] running on GCE node
I20260619 06:53:31.213161 30071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:31.213891 30071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:31.215063 30071 hybrid_clock.cc:648] HybridClock initialized: now 1781852011215049 us; error 27 us; skew 500 ppm
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:31.218116 30071 init.cc:377] Logged in from keytab as oryx/127.22.113.2@KRBTEST.COM (short username oryx)
I20260619 06:53:31.219089 30071 webserver.cc:460] Webserver started at http://127.22.113.2:39847/ using document root <none> and password file <none>
I20260619 06:53:31.219782 30071 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:31.219898 30071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:31.224363 30071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:31.226660 30086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.228197 30071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:31.228348 30071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "375b00f6d236497baea694a3c907dea8"
format_stamp: "Formatted at 2026-06-19 06:53:29 on dist-test-slave-g8zg"
server_key: "1233fcd961d34d7da4c38ccfd147b56d"
server_key_iv: "37cc007e1cdc39f9df5f5dcee7137cee"
server_key_version: "encryptionkey@0"
I20260619 06:53:31.228956 30071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:31.256470 30071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:31.260471 30071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:31.260687 30071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:31.261304 30071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:31.262279 30071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:31.262351 30071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.262418 30071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:31.262466 30071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.272531 30071 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:36353
I20260619 06:53:31.272622 30199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:36353 every 8 connection(s)
I20260619 06:53:31.273690 30071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:31.277716 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30071
I20260619 06:53:31.277961 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29430
I20260619 06:53:31.285969 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:45993
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:34219
--webserver_port=39811
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:31.295954 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.277520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:43653 (local address 127.22.113.61:42915)
0619 06:53:31.277645 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.277650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.277802 (+ 152us) server_negotiation.cc:407] Connection header received
0619 06:53:31.279846 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.279849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.279904 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.279958 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.282088 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.282943 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285404 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.285746 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285820 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.290995 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.291000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.291005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.291050 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.292899 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293757 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.293761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.293762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.293970 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.294281 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.294286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.294287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.294551 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.294628 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.295589 (+ 961us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.295755 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:53:31.295954 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.275819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:43663 (local address 127.22.113.60:38709)
0619 06:53:31.275968 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.275972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.277156 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 06:53:31.279846 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.279849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.279904 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.279958 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.282151 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.283121 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285225 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.285601 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.290068 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.290072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.290075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.290106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.292876 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293753 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.293757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.293759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.293859 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.294297 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.294300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.294302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.294542 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.294628 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.295598 (+ 970us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.295755 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:53:31.296041 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.276610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:43619 (local address 127.22.113.62:34219)
0619 06:53:31.276803 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.276809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.281276 (+ 4467us) server_negotiation.cc:407] Connection header received
0619 06:53:31.281453 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.281456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.281513 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.281568 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.282188 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.282958 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285289 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.285646 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285744 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.290509 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.290514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.290518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.290553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.292701 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293706 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.293711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.293714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.293818 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.294300 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.294307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.294312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.294594 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.294693 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.295611 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.295777 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:53:31.298448 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.276334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.22.113.2:43653)
0619 06:53:31.277686 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.277701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.279557 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.280368 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.280378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.280984 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.281918 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.281936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.283140 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.283144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.285294 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.285303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.286102 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.286106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.286208 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.287413 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.287428 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.289454 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.293027 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.293033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.293626 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.294064 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.294068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.294070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.294170 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.294708 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.294717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.295222 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.296403 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.297019 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":772,"spinlock_wait_cycles":7808,"thread_start_us":204,"threads_started":1}
I20260619 06:53:31.298445 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.276342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:43619)
0619 06:53:31.281178 (+ 4836us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.281193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.281346 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.281718 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.281721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.281910 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.282054 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.282068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.283140 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.283144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.285114 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.285131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.285960 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.286361 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.287272 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.287293 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.290282 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.292863 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.292872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.292894 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.293556 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.293945 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.293954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.294170 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.294764 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.294769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.295226 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.296343 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.296909 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4761,"mutex_wait_us":1460,"spinlock_wait_cycles":1920,"thread_start_us":130,"threads_started":1}
I20260619 06:53:31.299749 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.276333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.22.113.2:43663)
0619 06:53:31.276948 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.277001 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.279557 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.280368 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.280378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.280984 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.281926 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.281946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.283239 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.283263 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.285055 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.285068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.285897 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.285903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.286285 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.287102 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.287126 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.289862 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.293027 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.293030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.293033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.293626 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.294097 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.294102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.294106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.294174 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.294708 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.294717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.295226 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.296570 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.297076 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":740,"spinlock_wait_cycles":19584,"thread_start_us":171,"threads_started":1}
I20260619 06:53:31.300179 30202 heartbeater.cc:347] Connected to a master server at 127.22.113.61:42915
I20260619 06:53:31.300496 30202 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.300475 30200 heartbeater.cc:347] Connected to a master server at 127.22.113.60:38709
I20260619 06:53:31.300560 30200 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.300179 30201 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:31.301066 30201 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.301210 30200 heartbeater.cc:511] Master 127.22.113.60:38709 requested a full tablet report, sending...
I20260619 06:53:31.301285 30202 heartbeater.cc:511] Master 127.22.113.61:42915 requested a full tablet report, sending...
I20260619 06:53:31.301210 30201 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:31.302734 29883 ts_manager.cc:194] Registered new tserver with Master: 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353)
I20260619 06:53:31.303134 29701 ts_manager.cc:194] Registered new tserver with Master: 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353)
I20260619 06:53:31.303687 29100 ts_manager.cc:194] Re-registered known tserver with Master: 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353)
I20260619 06:53:31.304584 29100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:43619
I20260619 06:53:31.307683 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.302594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:41832)
0619 06:53:31.302743 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.302760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.302854 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.304039 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.304042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.304061 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.304206 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.304213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.305075 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.305080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.306203 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.306213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.307340 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.307343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.307422 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.307443 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.307497 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.307564 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260619 06:53:31.307739 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.303544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.22.113.60:38709)
0619 06:53:31.303787 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.303795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.303808 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:53:31.303852 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.303855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.303926 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.304024 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.304322 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.304968 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.306378 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.307164 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.307325 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.307496 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.307535 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.307609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260619 06:53:31.308686 29894 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 06:53:31.309185 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4525063167e4d0ba9fabf6f58769e67. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:31.309346 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: LEARNER
W20260619 06:53:31.414582 30207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:31.414839 30207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:31.414881 30207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:31.419023 30207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:31.419087 30207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:31.419147 30207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:31.424852 30207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:45993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=39811
--tserver_master_addrs=127.22.113.62:34219,127.22.113.61:42915,127.22.113.60:38709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:31.425866 30207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:31.426851 30207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:31.437459 30212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:31.437413 30215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:31.437423 30213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:31.437702 30207 server_base.cc:1034] running on GCE node
I20260619 06:53:31.438210 30207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:31.438827 30207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:31.440027 30207 hybrid_clock.cc:648] HybridClock initialized: now 1781852011440009 us; error 41 us; skew 500 ppm
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:31.442945 30207 init.cc:377] Logged in from keytab as oryx/127.22.113.3@KRBTEST.COM (short username oryx)
I20260619 06:53:31.443915 30207 webserver.cc:460] Webserver started at http://127.22.113.3:39811/ using document root <none> and password file <none>
I20260619 06:53:31.444504 30207 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:31.444581 30207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:31.450268 30207 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:31.452823 30222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.454373 30207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:31.454514 30207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "e1f4bda6552f4989a145d30cf73f8085"
format_stamp: "Formatted at 2026-06-19 06:53:30 on dist-test-slave-g8zg"
server_key: "b2d964484a461d7e5ab8cc0a6406e2cb"
server_key_iv: "1c526357ee9e0cc1d76a9d820ed59012"
server_key_version: "encryptionkey@0"
I20260619 06:53:31.455367 30207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:31.477691 30207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:31.480705 30207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:31.480901 30207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:31.481472 30207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:31.482312 30207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:31.482385 30207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.482455 30207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:31.482507 30207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:31.492961 30207 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:45993
I20260619 06:53:31.493027 30335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:45993 every 8 connection(s)
I20260619 06:53:31.494004 30207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:31.495538 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30207
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.60@KRBTEST.COM
Jun 19 06:53:31 dist-test-slave-g8zg krb5kdc[29055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852011, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.61@KRBTEST.COM
I20260619 06:53:31.514338 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.498656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.22.113.60:38709)
0619 06:53:31.498877 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.498883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.504633 (+ 5750us) server_negotiation.cc:407] Connection header received
0619 06:53:31.504723 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.504726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.504779 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.504868 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.505905 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.506748 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.508038 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.508419 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.508493 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.510341 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.510345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.510348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.510374 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.512305 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.512876 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.512882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.512884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.512965 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.513312 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.513318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.513321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.513550 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.513669 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.514039 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.514171 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260619 06:53:31.515354 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.496336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:35145 (local address 127.22.113.62:34219)
0619 06:53:31.496640 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.496645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.497787 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 06:53:31.499134 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.499139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.499205 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.499295 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.501643 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502475 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.504529 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.504854 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.504946 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.510852 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.510857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.510862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.510893 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.512530 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.513182 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.513201 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.513203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.513287 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.513774 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.513777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.513779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.513999 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.514063 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.515047 (+ 984us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.515181 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260619 06:53:31.516961 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.498587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:52987 (local address 127.22.113.61:42915)
0619 06:53:31.498718 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.498724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.499676 (+ 952us) server_negotiation.cc:407] Connection header received
0619 06:53:31.499860 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.499863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.499921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.500011 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.501647 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502474 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.505568 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.505928 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.506024 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.511015 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.511021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.511029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.511061 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.514533 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.515299 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.515305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.515309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.515476 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.515795 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.515798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.515803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.516051 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.516139 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.516586 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.516722 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:53:31.517459 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.498274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.22.113.62:34219)
0619 06:53:31.498490 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.498504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.500809 (+ 2305us) server_negotiation.cc:407] Connection header received
0619 06:53:31.501018 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.501021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.501072 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.501139 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.502269 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.503042 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.504745 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.505069 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.505141 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.508079 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.508086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.508089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.508125 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.510612 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.516044 (+ 5432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.516048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.516050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.516146 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.516540 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.516543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.516545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.516743 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.516857 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.517190 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.517317 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260619 06:53:31.517846 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.498198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.22.113.3:52987)
0619 06:53:31.499568 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.499584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.499740 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.500143 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.500146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.500591 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.501451 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.501474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502669 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.502672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.505365 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.505374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.506501 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.506507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.506627 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.507666 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.507684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.510853 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.514873 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.514876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.514878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.515138 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.515598 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.515601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.515603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.515698 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.516207 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.516211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.516325 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.517428 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.517603 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":1273,"spinlock_wait_cycles":79104,"thread_start_us":102,"threads_started":1}
I20260619 06:53:31.517932 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.496969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:35145)
0619 06:53:31.497594 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.497650 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.498842 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.499766 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.499777 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.500585 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.501451 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.501474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502669 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.502672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.504411 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.504420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.505113 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.505119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.505422 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.506176 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.506197 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.510694 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.512654 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.512661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.512678 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.513038 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.513401 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.513404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.513406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.513665 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.514160 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.514166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.514769 (+ 603us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.515758 (+ 989us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.516135 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":458,"spinlock_wait_cycles":24320,"thread_start_us":172,"threads_started":1}
I20260619 06:53:31.519297 30337 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34219
I20260619 06:53:31.519299 30338 heartbeater.cc:347] Connected to a master server at 127.22.113.61:42915
I20260619 06:53:31.519585 30337 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.519584 30338 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.519966 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.499062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.22.113.61:42915)
0619 06:53:31.499984 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.499988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.504266 (+ 4278us) server_negotiation.cc:407] Connection header received
0619 06:53:31.508419 (+ 4153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.508425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.508485 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.508584 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.509544 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.510416 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.512945 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.513353 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.513454 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.515730 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.515737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.515741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.515770 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.517523 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.518076 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.518079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.518081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.518164 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.518453 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.518456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.518457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.518650 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.518772 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.519529 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.519802 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":66,"threads_started":1}
I20260619 06:53:31.520282 30338 heartbeater.cc:511] Master 127.22.113.61:42915 requested a full tablet report, sending...
I20260619 06:53:31.520458 30337 heartbeater.cc:511] Master 127.22.113.62:34219 requested a full tablet report, sending...
I20260619 06:53:31.521430 29100 ts_manager.cc:194] Re-registered known tserver with Master: e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993)
I20260619 06:53:31.522003 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.496480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:59647 (local address 127.22.113.60:38709)
0619 06:53:31.496737 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.496741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.498970 (+ 2229us) server_negotiation.cc:407] Connection header received
0619 06:53:31.499152 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.499155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.499214 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.499295 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:31.501647 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502494 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.504553 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.504904 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.504978 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.509763 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:31.509770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:31.509774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:31.509803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:31.513173 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.514656 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.514661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.514663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.514754 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:31.520972 (+ 6218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:31.520976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:31.520978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:31.521224 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:31.521302 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.521698 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.521840 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260619 06:53:31.522070 29100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:35145
I20260619 06:53:31.522569 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.496969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.22.113.3:59647)
0619 06:53:31.498874 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.498888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.499005 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.499766 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.499777 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.500581 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:31.501451 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.501475 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.502670 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.502673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.504417 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.504427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.505158 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.505162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.506164 (+ 1002us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.506872 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:31.506892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:31.509556 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:31.514112 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.514116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.514120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.514400 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.517220 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:31.517224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:31.517226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:31.517325 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:31.521409 (+ 4084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:31.521412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:31.521508 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.522203 (+ 695us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.522384 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":1283,"thread_start_us":136,"threads_started":1}
I20260619 06:53:31.522738 29701 ts_manager.cc:194] Registered new tserver with Master: e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993)
I20260619 06:53:31.523226 30336 heartbeater.cc:347] Connected to a master server at 127.22.113.60:38709
I20260619 06:53:31.523358 30336 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:31.523525 30336 heartbeater.cc:511] Master 127.22.113.60:38709 requested a full tablet report, sending...
I20260619 06:53:31.520879 29097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:53:31.524266 29883 ts_manager.cc:194] Registered new tserver with Master: e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993)
W20260619 06:53:31.525031 29097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:31.527815 29848 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: NON_VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: true } } }
I20260619 06:53:31.527949 29848 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:31.531808 30355 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: attempting to promote NON_VOTER fb8e18be3f274c4ea3ed6bb9e7e7adb9 to VOTER
I20260619 06:53:31.532217 30355 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } }
I20260619 06:53:31.533464 29711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:53:31.533645 29894 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 LEARNER]: Refusing update from remote peer e4525063167e4d0ba9fabf6f58769e67: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:53:31.533824 30354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 06:53:31.534263 30354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 06:53:31.535998 30354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.537632 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.537752 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:31.537886 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.537986 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:31.538200 30354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.538293 30354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:31.538064 29894 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.539978 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.540102 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:31.539106 29711 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.544966 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4525063167e4d0ba9fabf6f58769e67" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:31.545125 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:31.545979 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.540175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36353 (local address 127.0.0.1:46332)
0619 06:53:31.540470 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.540486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.540568 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.541539 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.541542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.541557 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.541659 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.541665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.543178 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.543181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.544439 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.544449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.545595 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.545624 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.545712 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.545740 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.545785 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.545850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260619 06:53:31.546861 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.540780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.22.113.2:36353)
0619 06:53:31.541078 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.541098 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.541120 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:53:31.541195 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.541200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.541393 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.541789 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.541892 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.543038 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.544616 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.545466 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.546420 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.546590 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.546641 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.546725 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":134,"threads_started":1}
I20260619 06:53:31.549813 30134 tablet_service.cc:1467] Processing CreateTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 (DEFAULT_TABLE table=test-table [id=9d201c0a40f746c7856e78ba6f4485f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:31.550911 30134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a4d2be473c45b9ba08cdd6e88b9da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:31.551743 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.542795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39269 (local address 127.0.0.1:38630)
0619 06:53:31.545420 (+ 2625us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.545441 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.545944 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.547042 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.547045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.547062 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.547173 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.547180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.548628 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.548631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.549771 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.549782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.551341 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.551345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.551436 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.551460 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.551529 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.551611 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2511,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260619 06:53:31.552418 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.543000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.22.113.1:39269)
0619 06:53:31.543461 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.543478 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.545876 (+ 2398us) server_negotiation.cc:407] Connection header received
0619 06:53:31.546129 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.546138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.546497 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.546695 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.547331 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.548486 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.549925 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.550936 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.551376 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.551662 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.551748 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.552198 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":78,"threads_started":1}
I20260619 06:53:31.555068 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.543158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:45993 (local address 127.0.0.1:50632)
0619 06:53:31.546166 (+ 3008us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.546181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.546273 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.548206 (+ 1933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.548209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.548225 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.548302 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.548307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.549433 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.549438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.550452 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.550463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.554626 (+ 4163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.554630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.554722 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.554752 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.554810 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.554895 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":104,"threads_started":1}
I20260619 06:53:31.556054 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.543812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.22.113.3:45993)
0619 06:53:31.544153 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.544165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.547759 (+ 3594us) server_negotiation.cc:407] Connection header received
0619 06:53:31.547853 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.547859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.548048 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.548443 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.548590 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.549303 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.550591 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.554475 (+ 3884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.555535 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.555706 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.555773 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.555881 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260619 06:53:31.556893 29997 tablet_service.cc:1467] Processing CreateTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 (DEFAULT_TABLE table=test-table [id=9d201c0a40f746c7856e78ba6f4485f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:31.557925 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a4d2be473c45b9ba08cdd6e88b9da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:31.560034 30370 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Bootstrap starting.
I20260619 06:53:31.559633 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 (DEFAULT_TABLE table=test-table [id=9d201c0a40f746c7856e78ba6f4485f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:31.560837 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a4d2be473c45b9ba08cdd6e88b9da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:31.562896 30370 tablet_bootstrap.cc:654] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:31.563973 30370 log.cc:824] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:31.566594 30370 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: No bootstrap required, opened a new log
I20260619 06:53:31.566818 30370 ts_tablet_manager.cc:1397] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 06:53:31.570355 30373 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Bootstrap starting.
I20260619 06:53:31.571161 30371 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Bootstrap starting.
I20260619 06:53:31.572021 30370 raft_consensus.cc:348] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.572326 30370 raft_consensus.cc:374] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:31.572383 30370 raft_consensus.cc:729] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375b00f6d236497baea694a3c907dea8, State: Initialized, Role: FOLLOWER
I20260619 06:53:31.572754 30373 tablet_bootstrap.cc:654] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:31.572935 30370 consensus_queue.cc:260] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.573073 30371 tablet_bootstrap.cc:654] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:31.573884 30373 log.cc:824] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:31.573959 30371 log.cc:824] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:31.574191 30370 ts_tablet_manager.cc:1428] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:53:31.575440 30201 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
I20260619 06:53:31.576412 30373 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: No bootstrap required, opened a new log
I20260619 06:53:31.576412 30371 tablet_bootstrap.cc:492] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: No bootstrap required, opened a new log
I20260619 06:53:31.576613 30373 ts_tablet_manager.cc:1397] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260619 06:53:31.576646 30371 ts_tablet_manager.cc:1397] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20260619 06:53:31.580413 30371 raft_consensus.cc:348] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.580752 30371 raft_consensus.cc:374] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:31.580833 30371 raft_consensus.cc:729] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6f245104c74b49ae9ba3828a81b947, State: Initialized, Role: FOLLOWER
I20260619 06:53:31.581174 30373 raft_consensus.cc:348] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.581297 30371 consensus_queue.cc:260] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.581409 30373 raft_consensus.cc:374] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:31.581475 30373 raft_consensus.cc:729] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f4bda6552f4989a145d30cf73f8085, State: Initialized, Role: FOLLOWER
I20260619 06:53:31.581987 30064 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
I20260619 06:53:31.582370 30371 ts_tablet_manager.cc:1428] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:53:31.582625 30373 consensus_queue.cc:260] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.583988 30337 heartbeater.cc:503] Master 127.22.113.62:34219 was elected leader, sending a full tablet report...
I20260619 06:53:31.586791 30373 ts_tablet_manager.cc:1428] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
W20260619 06:53:31.745450 30339 tablet.cc:2369] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:31.775681 30203 tablet.cc:2369] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:53:31.786415 30066 tablet.cc:2369] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:31.937958 30374 raft_consensus.cc:482] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:31.938186 30374 raft_consensus.cc:504] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.939307 30374 leader_election.cc:288] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993), bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:31.943627 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.939904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:46959 (local address 127.22.113.1:39269)
0619 06:53:31.940043 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.940049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.940211 (+ 162us) server_negotiation.cc:407] Connection header received
0619 06:53:31.940402 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.940405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.940446 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.940555 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.940936 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.941464 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.942589 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.942990 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.943113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.943168 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.943429 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.943494 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:53:31.943657 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.939642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39269 (local address 127.22.113.2:46959)
0619 06:53:31.940086 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.940103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.940267 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.940589 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.940595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.940615 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.940786 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.940792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.941584 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.941589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.942456 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.942463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.943112 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.943116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.943267 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.943319 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.943413 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.943509 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":186,"thread_start_us":57,"threads_started":1}
I20260619 06:53:31.943773 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.939766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:37509 (local address 127.22.113.3:45993)
0619 06:53:31.940087 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:31.940094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:31.940119 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:53:31.940338 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:31.940342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:31.940403 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:31.940473 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:31.940910 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.941559 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.942649 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.943063 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.943197 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.943297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:31.943503 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 06:53:31.943573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260619 06:53:31.944164 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:31.939642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:45993 (local address 127.22.113.2:37509)
0619 06:53:31.940014 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:31.940032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:31.940181 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:31.940550 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:31.940553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:31.940571 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:31.940781 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.940787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.941684 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.941690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.942519 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:31.942526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:31.943195 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:31.943198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:31.943373 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:31.943393 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:53:31.943487 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:53:31.943573 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":105,"thread_start_us":95,"threads_started":1}
I20260619 06:53:31.944782 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a4d2be473c45b9ba08cdd6e88b9da1" candidate_uuid: "375b00f6d236497baea694a3c907dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f4bda6552f4989a145d30cf73f8085" is_pre_election: true
I20260619 06:53:31.944785 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a4d2be473c45b9ba08cdd6e88b9da1" candidate_uuid: "375b00f6d236497baea694a3c907dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6f245104c74b49ae9ba3828a81b947" is_pre_election: true
I20260619 06:53:31.945161 30017 raft_consensus.cc:2454] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 375b00f6d236497baea694a3c907dea8 in term 0.
I20260619 06:53:31.945161 30290 raft_consensus.cc:2454] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 375b00f6d236497baea694a3c907dea8 in term 0.
I20260619 06:53:31.945694 30089 leader_election.cc:302] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375b00f6d236497baea694a3c907dea8, e1f4bda6552f4989a145d30cf73f8085; no voters:
I20260619 06:53:31.947518 30374 raft_consensus.cc:2781] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:31.947618 30374 raft_consensus.cc:482] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:31.947660 30374 raft_consensus.cc:3037] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:31.948984 30374 raft_consensus.cc:504] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.949440 30374 leader_election.cc:288] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [CANDIDATE]: Term 1 election: Requested vote from peers e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993), bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:31.949913 30290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a4d2be473c45b9ba08cdd6e88b9da1" candidate_uuid: "375b00f6d236497baea694a3c907dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f4bda6552f4989a145d30cf73f8085"
I20260619 06:53:31.949911 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a4d2be473c45b9ba08cdd6e88b9da1" candidate_uuid: "375b00f6d236497baea694a3c907dea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6f245104c74b49ae9ba3828a81b947"
I20260619 06:53:31.950067 30017 raft_consensus.cc:3037] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:31.950074 30290 raft_consensus.cc:3037] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:31.951391 30290 raft_consensus.cc:2454] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 375b00f6d236497baea694a3c907dea8 in term 1.
I20260619 06:53:31.951519 30017 raft_consensus.cc:2454] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 375b00f6d236497baea694a3c907dea8 in term 1.
I20260619 06:53:31.951740 30089 leader_election.cc:302] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375b00f6d236497baea694a3c907dea8, e1f4bda6552f4989a145d30cf73f8085; no voters:
I20260619 06:53:31.951941 30374 raft_consensus.cc:2781] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:31.952212 30374 raft_consensus.cc:686] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 1 LEADER]: Becoming Leader. State: Replica: 375b00f6d236497baea694a3c907dea8, State: Running, Role: LEADER
I20260619 06:53:31.953110 30374 consensus_queue.cc:237] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } }
I20260619 06:53:31.958204 29097 catalog_manager.cc:5495] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 375b00f6d236497baea694a3c907dea8 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "375b00f6d236497baea694a3c907dea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:32.005888 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:32.001958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.22.113.2:36353)
0619 06:53:32.002134 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:32.002139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:32.002157 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:32.002264 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:32.002266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:32.002331 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:32.002470 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:32.002731 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.003360 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.004235 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.004462 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.004571 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:32.004928 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:32.005465 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:32.005607 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:32.005643 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 06:53:32.005704 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:53:32.013023 30290 raft_consensus.cc:1261] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Refusing update from remote peer 375b00f6d236497baea694a3c907dea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:32.013023 30017 raft_consensus.cc:1261] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Refusing update from remote peer 375b00f6d236497baea694a3c907dea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:32.014034 30374 consensus_queue.cc:1035] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:32.014312 30381 consensus_queue.cc:1035] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:32.023742 30385 mvcc.cc:205] Tried to move back new op lower bound from 7298465841195806720 to 7298465840965033984. Current Snapshot: MvccSnapshot[applied={T|T < 7298465841195806720}]
I20260619 06:53:32.028661 30386 mvcc.cc:205] Tried to move back new op lower bound from 7298465841195806720 to 7298465840965033984. Current Snapshot: MvccSnapshot[applied={T|T < 7298465841195806720}]
I20260619 06:53:32.039625 29100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:32.039848 29100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:32.045488 29100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4525063167e4d0ba9fabf6f58769e67: Sending DeleteTablet for 3 replicas of tablet 89a4d2be473c45b9ba08cdd6e88b9da1
I20260619 06:53:32.046670 30134 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:32 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:46332
I20260619 06:53:32.046865 29997 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:32 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:38630
I20260619 06:53:32.047809 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a4d2be473c45b9ba08cdd6e88b9da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:32 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:50632
I20260619 06:53:32.048754 30394 tablet_replica.cc:330] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: stopping tablet replica
I20260619 06:53:32.049415 30393 tablet_replica.cc:330] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: stopping tablet replica
I20260619 06:53:32.049445 30394 raft_consensus.cc:2229] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:32.049768 30393 raft_consensus.cc:2229] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:32.049855 30394 raft_consensus.cc:2258] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:32.050207 30393 raft_consensus.cc:2258] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:32.050558 30395 tablet_replica.cc:330] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: stopping tablet replica
I20260619 06:53:32.050717 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29068
I20260619 06:53:32.051785 30395 raft_consensus.cc:2229] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:32.052127 30395 raft_consensus.cc:2258] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:32.053365 30393 ts_tablet_manager.cc:1905] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:32.056214 30393 ts_tablet_manager.cc:1918] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:32.056313 30393 log.cc:1196] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/89a4d2be473c45b9ba08cdd6e88b9da1
I20260619 06:53:32.056656 30393 ts_tablet_manager.cc:1939] T 89a4d2be473c45b9ba08cdd6e88b9da1 P 375b00f6d236497baea694a3c907dea8: Deleting consensus metadata
I20260619 06:53:32.056664 30395 ts_tablet_manager.cc:1905] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:32.059327 30395 ts_tablet_manager.cc:1918] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:32.059427 30395 log.cc:1196] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/89a4d2be473c45b9ba08cdd6e88b9da1
I20260619 06:53:32.059798 30395 ts_tablet_manager.cc:1939] T 89a4d2be473c45b9ba08cdd6e88b9da1 P e1f4bda6552f4989a145d30cf73f8085: Deleting consensus metadata
I20260619 06:53:32.060529 30394 ts_tablet_manager.cc:1905] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:53:32.060992 30224 connection.cc:535] server connection from 127.0.0.1:50632 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 06:53:32.061048 30087 connection.cc:535] server connection from 127.0.0.1:46332 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:53:32.065582 30394 ts_tablet_manager.cc:1918] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:32.065702 30394 log.cc:1196] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/89a4d2be473c45b9ba08cdd6e88b9da1
I20260619 06:53:32.066052 30394 ts_tablet_manager.cc:1939] T 89a4d2be473c45b9ba08cdd6e88b9da1 P bc6f245104c74b49ae9ba3828a81b947: Deleting consensus metadata
W20260619 06:53:32.067840 29952 connection.cc:441] server connection from 127.0.0.1:38630 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38630 (request call id 1) could send its response
I20260619 06:53:32.076004 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:32.049019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.22.113.61:42915)
0619 06:53:32.055781 (+ 6762us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:32.055789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:32.058081 (+ 2292us) server_negotiation.cc:407] Connection header received
0619 06:53:32.058266 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:32.058272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:32.058353 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:32.058466 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:32.059936 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.060832 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.062118 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.062498 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.062596 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:32.069925 (+ 7329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:32.069929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:32.069932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:32.069959 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:32.072612 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.073437 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.073442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.073444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.073528 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.073860 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.073862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.073863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.074129 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:32.074276 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:32.075613 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0619 06:53:32.075769 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6670,"thread_start_us":98,"threads_started":1}
I20260619 06:53:32.076635 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:32.065513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.22.113.60:38709)
0619 06:53:32.065973 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:32.065980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:32.066890 (+ 910us) server_negotiation.cc:407] Connection header received
0619 06:53:32.066942 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:32.066945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:32.066997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:32.067102 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:32.068078 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.068975 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.070186 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.070531 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.070602 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:32.072565 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:32.072569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:32.072572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:32.072596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:32.074728 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.075263 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.075267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.075269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.075350 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.075647 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.075650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.075652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.075857 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:32.075958 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:32.076327 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 06:53:32.076455 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":74,"threads_started":1}
I20260619 06:53:32.077139 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:32.065078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.22.113.61:42915)
0619 06:53:32.065394 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:32.065400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:32.066486 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 06:53:32.066668 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:32.066671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:32.066734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:32.066801 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:32.068756 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.069630 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.070985 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.071426 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.071513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:32.073288 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:32.073293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:32.073296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:32.073321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:32.075183 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.075687 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.075690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.075692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.075768 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.076067 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.076070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.076072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.076347 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:32.076438 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:32.076858 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 06:53:32.076989 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260619 06:53:32.086524 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:32.049277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.22.113.60:38709)
0619 06:53:32.055598 (+ 6321us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:32.055605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:32.075199 (+ 19594us) server_negotiation.cc:407] Connection header received
0619 06:53:32.075291 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:32.075296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:32.075357 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:32.075450 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:32.078884 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.079585 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.080370 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:32.080641 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:32.080741 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:32.082639 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:32.082643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:32.082646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:32.082671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:32.084418 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.084836 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.084842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.084844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.084928 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:32.085206 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:32.085209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:32.085211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:32.085447 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:32.085576 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:32.086211 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 06:53:32.086361 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6196,"thread_start_us":85,"threads_started":1}
I20260619 06:53:33.027972 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.027336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:50447)
0619 06:53:33.027703 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.027813 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
W20260619 06:53:33.028676 30337 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:34219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.030123 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.029580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:53365)
0619 06:53:33.029872 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.029993 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
W20260619 06:53:33.030814 30201 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:34219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.031590 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.030928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:54073)
0619 06:53:33.031402 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.031491 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":77,"threads_started":1}
W20260619 06:53:33.031929 30064 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:34219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.545991 30411 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4525063167e4d0ba9fabf6f58769e67)
I20260619 06:53:33.546186 30411 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } }
I20260619 06:53:33.546519 30412 raft_consensus.cc:482] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e4525063167e4d0ba9fabf6f58769e67)
I20260619 06:53:33.546759 30412 raft_consensus.cc:504] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } }
I20260619 06:53:33.546859 30411 leader_election.cc:288] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219), b01fd4ae0d8e45e085983ed02686defb (127.22.113.61:42915)
I20260619 06:53:33.547724 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.547145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44516)
0619 06:53:33.547494 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.547593 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
W20260619 06:53:33.547912 29871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.548527 30412 leader_election.cc:288] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219), fb8e18be3f274c4ea3ed6bb9e7e7adb9 (127.22.113.60:38709)
I20260619 06:53:33.548861 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.548606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44532)
0619 06:53:33.548745 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.548795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:53:33.548914 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.548269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44524)
0619 06:53:33.548719 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.548812 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":55,"threads_started":1}
W20260619 06:53:33.548995 29871 leader_election.cc:334] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219): Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
W20260619 06:53:33.549080 29688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.549901 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.549677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44540)
0619 06:53:33.549771 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.549822 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 06:53:33.550019 29688 leader_election.cc:334] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219): Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.551975 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.547222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.22.113.61:42915)
0619 06:53:33.547543 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.547548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.547568 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:53:33.547742 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.547747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.547815 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.547930 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.548357 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.549454 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.550815 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.551405 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.551522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.551593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.551765 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.551861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260619 06:53:33.551992 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.547126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:42915 (local address 127.0.0.1:45384)
0619 06:53:33.547473 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.547491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.547607 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.548044 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.548047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.548067 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.548214 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.548225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.549603 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.549606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.550692 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.550701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.551533 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.551537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.551629 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.551661 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.551751 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.551835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":59,"threads_started":1}
I20260619 06:53:33.552455 29711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b01fd4ae0d8e45e085983ed02686defb" is_pre_election: true
I20260619 06:53:33.552675 29711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb8e18be3f274c4ea3ed6bb9e7e7adb9 in term 1.
I20260619 06:53:33.553005 29869 leader_election.cc:302] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01fd4ae0d8e45e085983ed02686defb, fb8e18be3f274c4ea3ed6bb9e7e7adb9; no voters: e4525063167e4d0ba9fabf6f58769e67
I20260619 06:53:33.553272 30411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:53:33.553345 30411 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e4525063167e4d0ba9fabf6f58769e67)
I20260619 06:53:33.553388 30411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:53:33.553774 30419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.548624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.22.113.60:38709)
0619 06:53:33.548836 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.548844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.549538 (+ 694us) server_negotiation.cc:407] Connection header received
0619 06:53:33.549673 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.549676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.549727 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.549812 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.550195 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.550782 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.552448 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.553081 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.553193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.553279 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.553597 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.553679 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260619 06:53:33.553793 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.548357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38709 (local address 127.0.0.1:36368)
0619 06:53:33.549428 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.549444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.549553 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.549915 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.549918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.549937 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.550023 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.550028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.550904 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.550910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.551962 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.551971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.553345 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.553349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.553455 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.553481 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.553569 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.553659 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":39,"thread_start_us":71,"threads_started":1}
I20260619 06:53:33.554615 30411 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } }
I20260619 06:53:33.554977 30411 leader_election.cc:288] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 election: Requested vote from peers e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219), b01fd4ae0d8e45e085983ed02686defb (127.22.113.61:42915)
I20260619 06:53:33.555341 29894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01fd4ae0d8e45e085983ed02686defb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" is_pre_election: true
I20260619 06:53:33.555374 29711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b01fd4ae0d8e45e085983ed02686defb"
I20260619 06:53:33.555477 29711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:53:33.555485 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.555123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44550)
0619 06:53:33.555320 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.555411 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 06:53:33.555573 29894 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01fd4ae0d8e45e085983ed02686defb in current term 2: Already voted for candidate fb8e18be3f274c4ea3ed6bb9e7e7adb9 in this term.
I20260619 06:53:33.556038 29687 leader_election.cc:302] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01fd4ae0d8e45e085983ed02686defb; no voters: e4525063167e4d0ba9fabf6f58769e67, fb8e18be3f274c4ea3ed6bb9e7e7adb9
I20260619 06:53:33.556149 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.555947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44560)
0619 06:53:33.556041 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.556092 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:53:33.556231 29871 leader_election.cc:334] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219): Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
I20260619 06:53:33.556756 29711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb8e18be3f274c4ea3ed6bb9e7e7adb9 in term 2.
I20260619 06:53:33.556869 30412 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:53:33.557112 29869 leader_election.cc:302] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b01fd4ae0d8e45e085983ed02686defb, fb8e18be3f274c4ea3ed6bb9e7e7adb9; no voters: e4525063167e4d0ba9fabf6f58769e67
I20260619 06:53:33.557309 30411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:53:33.557543 30411 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [term 2 LEADER]: Becoming Leader. State: Replica: fb8e18be3f274c4ea3ed6bb9e7e7adb9, State: Running, Role: LEADER
I20260619 06:53:33.557827 30411 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } }
I20260619 06:53:33.559038 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb8e18be3f274c4ea3ed6bb9e7e7adb9. Latest consensus state: current_term: 2 leader_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:33.559229 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:33.559580 30423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:33.562692 30423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:33.563110 30423 catalog_manager.cc:1259] Loaded cluster ID: a698d9a104c3492490b3cc3f5fe6081e
I20260619 06:53:33.563158 30423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:33.563827 30423 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:33.564242 30423 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Loaded TSK: 0
I20260619 06:53:33.565167 30423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:33.599040 29884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:33.602280 29884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:33.605684 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.605388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44580)
0619 06:53:33.605528 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.605616 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:53:33.605837 29711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [term 2 FOLLOWER]: Refusing update from remote peer fb8e18be3f274c4ea3ed6bb9e7e7adb9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 06:53:33.606447 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.606219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.0.0.1:44584)
0619 06:53:33.606334 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.606387 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 06:53:33.606520 30411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 06:53:33.606701 29871 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 -> Peer e4525063167e4d0ba9fabf6f58769e67 (127.22.113.62:34219): Couldn't send request to peer e4525063167e4d0ba9fabf6f58769e67. Status: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:53:33.608332 30412 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb8e18be3f274c4ea3ed6bb9e7e7adb9. Latest consensus state: current_term: 2 leader_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:33.608489 30412 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:33.609650 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:33.609776 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:33.609977 30425 mvcc.cc:205] Tried to move back new op lower bound from 7298465847722921984 to 7298465847534964736. Current Snapshot: MvccSnapshot[applied={T|T < 7298465847722921984}]
I20260619 06:53:33.610227 30412 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e4525063167e4d0ba9fabf6f58769e67" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34219 } } peers { permanent_uuid: "b01fd4ae0d8e45e085983ed02686defb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 42915 } attrs { promote: false } } peers { permanent_uuid: "fb8e18be3f274c4ea3ed6bb9e7e7adb9" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38709 } attrs { promote: false } } }
I20260619 06:53:33.610370 30412 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01fd4ae0d8e45e085983ed02686defb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:53:33.620237 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.616127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36353 (local address 127.0.0.1:38236)
0619 06:53:33.616257 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.616276 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.616396 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.616984 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.616987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.617003 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.617095 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.617101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.617971 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.617977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.618907 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.618920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.619830 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.619836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.619941 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.619971 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.620022 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.620112 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:53:33.620821 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.616471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.22.113.2:36353)
0619 06:53:33.616750 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.616755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.616771 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:33.616828 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.616833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.616888 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.616978 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.617211 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.617875 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.619048 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.619702 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.620514 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.620578 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.620621 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.620690 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260619 06:53:33.621343 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.616409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39269 (local address 127.0.0.1:37830)
0619 06:53:33.616655 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.616667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.616752 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.617483 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.617486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.617505 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.617580 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.617585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.618746 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.618749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.620012 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.620022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.620960 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.620963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.621058 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.621091 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.621154 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.621234 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":24}
I20260619 06:53:33.621316 30134 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f90033d094898bafdb1f00ba8216c (DEFAULT_TABLE table=test-table [id=e348b082673344aea744b4fafeb454f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:33.621625 30134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f90033d094898bafdb1f00ba8216c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:33.621836 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.616961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.22.113.1:39269)
0619 06:53:33.617210 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.617216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.617232 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:33.617294 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.617297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.617348 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.617452 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.617721 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.618590 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.620169 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.620808 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.621533 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.621599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.621638 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.621716 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260619 06:53:33.622242 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.617313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.22.113.3:45993)
0619 06:53:33.617925 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.617930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.617943 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:53:33.618004 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.618007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.618056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.618149 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.618455 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.619290 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.620643 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.621211 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.621384 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.621615 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.621824 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.622066 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":100,"threads_started":1}
I20260619 06:53:33.622288 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.616648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:45993 (local address 127.0.0.1:49298)
0619 06:53:33.617600 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.617613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.617727 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.618180 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.618183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.618202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.618289 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.618294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.619430 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.619433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.620538 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.620546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.621379 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.621588 (+ 209us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.621695 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.621722 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.621807 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.621889 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260619 06:53:33.622399 29997 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f90033d094898bafdb1f00ba8216c (DEFAULT_TABLE table=test-table [id=e348b082673344aea744b4fafeb454f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:33.622668 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f90033d094898bafdb1f00ba8216c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:33.623070 30270 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f90033d094898bafdb1f00ba8216c (DEFAULT_TABLE table=test-table [id=e348b082673344aea744b4fafeb454f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:33.623453 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f90033d094898bafdb1f00ba8216c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:33.625303 30434 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947: Bootstrap starting.
I20260619 06:53:33.626380 30434 tablet_bootstrap.cc:654] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:33.626470 30435 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Bootstrap starting.
I20260619 06:53:33.626928 30433 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Bootstrap starting.
I20260619 06:53:33.627449 30435 tablet_bootstrap.cc:654] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:33.627985 30433 tablet_bootstrap.cc:654] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:33.628544 30434 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947: No bootstrap required, opened a new log
I20260619 06:53:33.628651 30434 ts_tablet_manager.cc:1397] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:33.628981 30435 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: No bootstrap required, opened a new log
I20260619 06:53:33.629060 30435 ts_tablet_manager.cc:1397] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:33.629215 30434 raft_consensus.cc:348] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.629354 30434 raft_consensus.cc:374] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:33.629393 30434 raft_consensus.cc:729] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6f245104c74b49ae9ba3828a81b947, State: Initialized, Role: FOLLOWER
I20260619 06:53:33.629415 30435 raft_consensus.cc:348] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.629558 30435 raft_consensus.cc:374] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:33.629534 30434 consensus_queue.cc:260] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.629594 30435 raft_consensus.cc:729] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f4bda6552f4989a145d30cf73f8085, State: Initialized, Role: FOLLOWER
I20260619 06:53:33.629742 30435 consensus_queue.cc:260] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.629895 30434 ts_tablet_manager.cc:1428] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:53:33.629931 30433 tablet_bootstrap.cc:492] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: No bootstrap required, opened a new log
I20260619 06:53:33.630070 30435 ts_tablet_manager.cc:1428] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:33.630090 30433 ts_tablet_manager.cc:1397] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:33.630656 30433 raft_consensus.cc:348] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.630853 30433 raft_consensus.cc:374] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:33.630963 30433 raft_consensus.cc:729] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375b00f6d236497baea694a3c907dea8, State: Initialized, Role: FOLLOWER
I20260619 06:53:33.631177 30433 consensus_queue.cc:260] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.631619 30433 ts_tablet_manager.cc:1428] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:33.631714 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.630744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:46187)
0619 06:53:33.631529 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.631598 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":170,"threads_started":1}
I20260619 06:53:33.633445 30336 heartbeater.cc:503] Master 127.22.113.60:38709 was elected leader, sending a full tablet report...
I20260619 06:53:33.633512 30063 heartbeater.cc:503] Master 127.22.113.60:38709 was elected leader, sending a full tablet report...
I20260619 06:53:33.633116 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.632212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:43063)
0619 06:53:33.632911 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.632976 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":108,"threads_started":1}
I20260619 06:53:33.634938 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.632732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:36471)
0619 06:53:33.634753 (+ 2021us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.634826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1951,"thread_start_us":121,"threads_started":1}
I20260619 06:53:33.635628 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.633975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:40561)
0619 06:53:33.635495 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.635543 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":103,"threads_started":1}
I20260619 06:53:33.635937 30200 heartbeater.cc:503] Master 127.22.113.60:38709 was elected leader, sending a full tablet report...
I20260619 06:53:33.636293 30440 raft_consensus.cc:482] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:33.636422 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.636084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:48449)
0619 06:53:33.636303 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.636356 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 06:53:33.636433 30440 raft_consensus.cc:504] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.637019 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.636137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:59349)
0619 06:53:33.636918 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.636966 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720}
I20260619 06:53:33.637985 30440 leader_election.cc:288] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353), bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:33.642302 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.638071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36353 (local address 127.22.113.3:60989)
0619 06:53:33.638227 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.638244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.638330 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.638886 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.638889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.638910 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.639037 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.639047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.640148 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.640151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.641076 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.641085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.641888 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.641891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.641992 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.642024 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.642103 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.642181 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":6}
I20260619 06:53:33.642371 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.638293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:60989 (local address 127.22.113.2:36353)
0619 06:53:33.638398 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.638402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.638414 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:33.638483 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.638486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.638534 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.638626 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.639168 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.640007 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.641212 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.641746 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.641844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.641884 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.642154 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.642235 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:53:33.642608 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.638800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:45943 (local address 127.22.113.1:39269)
0619 06:53:33.638897 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.638902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.638912 (+ 10us) server_negotiation.cc:407] Connection header received
0619 06:53:33.638951 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.638954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.638997 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.639079 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:33.639584 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.640274 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.641501 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.641998 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.642123 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.642170 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.642407 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.642501 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:53:33.642841 30154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f90033d094898bafdb1f00ba8216c" candidate_uuid: "e1f4bda6552f4989a145d30cf73f8085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375b00f6d236497baea694a3c907dea8" is_pre_election: true
I20260619 06:53:33.643095 30154 raft_consensus.cc:2454] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f4bda6552f4989a145d30cf73f8085 in term 0.
I20260619 06:53:33.643283 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.638302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39269 (local address 127.22.113.3:45943)
0619 06:53:33.638561 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.638577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.638663 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.639143 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:33.639149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:33.639188 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:33.639421 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.639430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.640388 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.640390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.641351 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:33.641361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.642148 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.642152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:33.642260 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.642285 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:53:33.642370 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:53:33.642456 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":26,"thread_start_us":111,"threads_started":1}
I20260619 06:53:33.643606 30226 leader_election.cc:302] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 375b00f6d236497baea694a3c907dea8, e1f4bda6552f4989a145d30cf73f8085; no voters:
I20260619 06:53:33.643891 30440 raft_consensus.cc:2781] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:33.643843 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f90033d094898bafdb1f00ba8216c" candidate_uuid: "e1f4bda6552f4989a145d30cf73f8085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6f245104c74b49ae9ba3828a81b947" is_pre_election: true
I20260619 06:53:33.643959 30440 raft_consensus.cc:482] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:33.644001 30440 raft_consensus.cc:3037] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:33.644001 30017 raft_consensus.cc:2454] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f4bda6552f4989a145d30cf73f8085 in term 0.
I20260619 06:53:33.645270 30440 raft_consensus.cc:504] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.645660 30440 leader_election.cc:288] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [CANDIDATE]: Term 1 election: Requested vote from peers 375b00f6d236497baea694a3c907dea8 (127.22.113.2:36353), bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269)
I20260619 06:53:33.646016 30154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f90033d094898bafdb1f00ba8216c" candidate_uuid: "e1f4bda6552f4989a145d30cf73f8085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375b00f6d236497baea694a3c907dea8"
I20260619 06:53:33.646162 30154 raft_consensus.cc:3037] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:33.646260 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f90033d094898bafdb1f00ba8216c" candidate_uuid: "e1f4bda6552f4989a145d30cf73f8085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6f245104c74b49ae9ba3828a81b947"
I20260619 06:53:33.646378 30017 raft_consensus.cc:3037] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:33.647547 30017 raft_consensus.cc:2454] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f4bda6552f4989a145d30cf73f8085 in term 1.
I20260619 06:53:33.647653 30154 raft_consensus.cc:2454] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f4bda6552f4989a145d30cf73f8085 in term 1.
I20260619 06:53:33.647866 30224 leader_election.cc:302] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc6f245104c74b49ae9ba3828a81b947, e1f4bda6552f4989a145d30cf73f8085; no voters:
I20260619 06:53:33.648020 30440 raft_consensus.cc:2781] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:33.648231 30440 raft_consensus.cc:686] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 1 LEADER]: Becoming Leader. State: Replica: e1f4bda6552f4989a145d30cf73f8085, State: Running, Role: LEADER
I20260619 06:53:33.648497 30440 consensus_queue.cc:237] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } }
I20260619 06:53:33.648892 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.648680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:36095)
0619 06:53:33.648780 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.648837 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 06:53:33.650818 29883 catalog_manager.cc:5495] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "e1f4bda6552f4989a145d30cf73f8085" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f4bda6552f4989a145d30cf73f8085" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 45993 } health_report { overall_health: HEALTHY } } }
I20260619 06:53:33.664906 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.660846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.22.113.3:45993)
0619 06:53:33.660997 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:33.661001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:33.661016 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:53:33.661212 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:33.661215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:33.661272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:33.661388 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:33.661640 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.662184 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.663203 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:33.663484 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:33.663595 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:33.663884 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:33.664474 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:33.664625 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:33.664689 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 06:53:33.664745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:53:33.668375 30017 raft_consensus.cc:1261] T 6f8f90033d094898bafdb1f00ba8216c P bc6f245104c74b49ae9ba3828a81b947 [term 1 FOLLOWER]: Refusing update from remote peer e1f4bda6552f4989a145d30cf73f8085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:33.668619 30154 raft_consensus.cc:1261] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Refusing update from remote peer e1f4bda6552f4989a145d30cf73f8085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:33.668993 30440 consensus_queue.cc:1035] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6f245104c74b49ae9ba3828a81b947" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:33.669231 30447 consensus_queue.cc:1035] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375b00f6d236497baea694a3c907dea8" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:33.671108 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.670783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:52089)
0619 06:53:33.670935 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.671011 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 06:53:33.672930 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.672625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:44145)
0619 06:53:33.672781 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.672845 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 06:53:33.674923 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.674619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:58423)
0619 06:53:33.674785 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.674844 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 06:53:33.675225 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.674851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.2:53705)
0619 06:53:33.675104 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.675160 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260619 06:53:33.676761 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.676531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.1:52301)
0619 06:53:33.676637 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.676691 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 06:53:33.677929 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.677290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34219 (local address 127.22.113.3:45961)
0619 06:53:33.677814 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.677864 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459}
I20260619 06:53:33.679738 30452 mvcc.cc:205] Tried to move back new op lower bound from 7298465847978840064 to 7298465847906942976. Current Snapshot: MvccSnapshot[applied={T|T < 7298465847978840064}]
I20260619 06:53:33.684257 29882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:33.684430 29882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:33.688746 29882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb8e18be3f274c4ea3ed6bb9e7e7adb9: Sending DeleteTablet for 3 replicas of tablet 6f8f90033d094898bafdb1f00ba8216c
I20260619 06:53:33.689409 30134 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f90033d094898bafdb1f00ba8216c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:33 UTC) from {username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:38236
I20260619 06:53:33.689894 30270 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f90033d094898bafdb1f00ba8216c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:33 UTC) from {username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:49298
I20260619 06:53:33.690264 30460 tablet_replica.cc:330] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: stopping tablet replica
I20260619 06:53:33.690394 30460 raft_consensus.cc:2229] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:33.690493 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29845
I20260619 06:53:33.690601 29997 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f90033d094898bafdb1f00ba8216c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:33 UTC) from {username='oryx', principal='oryx/127.22.113.60@KRBTEST.COM'} at 127.0.0.1:37830
I20260619 06:53:33.692286 30460 raft_consensus.cc:2258] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:33.692965 30460 ts_tablet_manager.cc:1905] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:33.695158 30460 ts_tablet_manager.cc:1918] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:33.695251 30460 log.cc:1196] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/6f8f90033d094898bafdb1f00ba8216c
I20260619 06:53:33.695542 30460 ts_tablet_manager.cc:1939] T 6f8f90033d094898bafdb1f00ba8216c P e1f4bda6552f4989a145d30cf73f8085: Deleting consensus metadata
I20260619 06:53:33.696220 29870 catalog_manager.cc:4841] TS e1f4bda6552f4989a145d30cf73f8085 (127.22.113.3:45993): tablet 6f8f90033d094898bafdb1f00ba8216c (table test-table [id=e348b082673344aea744b4fafeb454f7]) successfully deleted
I20260619 06:53:33.699455 30459 tablet_replica.cc:330] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: stopping tablet replica
I20260619 06:53:33.699628 30459 raft_consensus.cc:2229] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:33.699796 30459 raft_consensus.cc:2258] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:33.700464 30459 ts_tablet_manager.cc:1905] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:53:33.701961 30430 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:33.701283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39269 (local address 127.0.0.1:37844)
0619 06:53:33.701479 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.701498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:33.701598 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:33.701816 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:39269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 06:53:33.702190 29869 catalog_manager.cc:4584] TS bc6f245104c74b49ae9ba3828a81b947 (127.22.113.1:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f8f90033d094898bafdb1f00ba8216c: Network error: Client connection negotiation failed: client connection to 127.22.113.1:39269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:53:33.702824 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30071
I20260619 06:53:33.702878 30459 ts_tablet_manager.cc:1918] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:33.702982 30459 log.cc:1196] T 6f8f90033d094898bafdb1f00ba8216c P 375b00f6d236497baea694a3c907dea8: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/6f8f90033d094898bafdb1f00ba8216c
I20260619 06:53:33.712754 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30207
I20260619 06:53:33.713071 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:33.712713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36353 (local address 127.0.0.1:38250)
0619 06:53:33.712897 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:33.712965 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 06:53:33.720005 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29564
I20260619 06:53:33.727198 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 29665
2026-06-19T06:53:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5787 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:33 dist-test-slave-g8zg krb5kdc[30463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:33 dist-test-slave-g8zg krb5kdc[30463](info): set up 2 sockets
Jun 19 06:53:33 dist-test-slave-g8zg krb5kdc[30463](info): commencing operation
krb5kdc: starting...
W20260619 06:53:34.779980 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:34 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab.
Entry for principal oryx/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab.
I20260619 06:53:34.914197 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41421
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab
--principal=oryx/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:35.039789 30476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:35.040117 30476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:35.040196 30476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:35.044317 30476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:35.044414 30476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:53:35.044447 30476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:35.044492 30476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:35.044528 30476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:35.044561 30476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:35.050712 30476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:41421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.62.keytab
--principal=oryx/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:35.052014 30476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:35.053233 30476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:35.063035 30481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.063043 30484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:35.063185 30476 server_base.cc:1034] running on GCE node
W20260619 06:53:35.063043 30482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:35.064174 30476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:35.065404 30476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:35.066605 30476 hybrid_clock.cc:648] HybridClock initialized: now 1781852015066560 us; error 65 us; skew 500 ppm
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:35.069813 30476 init.cc:377] Logged in from keytab as oryx/127.22.113.62@KRBTEST.COM (short username oryx)
I20260619 06:53:35.070987 30476 webserver.cc:460] Webserver started at http://127.22.113.62:42501/ using document root <none> and password file <none>
I20260619 06:53:35.071749 30476 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:35.071825 30476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:35.072022 30476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:35.074378 30476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "f19f50a85faf44438b02769f4abedd21"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7b30b51bb0ab30cab6bb699b371c9d3f"
server_key_iv: "d68169322b6947a66c1b4d868e6b3cd9"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.074954 30476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "f19f50a85faf44438b02769f4abedd21"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7b30b51bb0ab30cab6bb699b371c9d3f"
server_key_iv: "d68169322b6947a66c1b4d868e6b3cd9"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.079398 30476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260619 06:53:35.082477 30491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.084187 30476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:35.084321 30476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "f19f50a85faf44438b02769f4abedd21"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7b30b51bb0ab30cab6bb699b371c9d3f"
server_key_iv: "d68169322b6947a66c1b4d868e6b3cd9"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.084427 30476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:35.108592 30476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:35.112036 30476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:35.112211 30476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:35.121403 30476 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:41421
I20260619 06:53:35.121469 30553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:41421 every 8 connection(s)
I20260619 06:53:35.122599 30476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:35.125783 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:35.131664 30554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Bootstrap starting.
I20260619 06:53:35.132180 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30476
I20260619 06:53:35.132325 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:35.132634 22980 external_mini_cluster.cc:1428] Setting key 511a9f319a811ae09c9143b11d36b715
I20260619 06:53:35.134711 30554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:35.135545 30554 log.cc:824] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:35.137425 30554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: No bootstrap required, opened a new log
I20260619 06:53:35.142114 30554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } }
I20260619 06:53:35.142376 30554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:35.142482 30554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19f50a85faf44438b02769f4abedd21, State: Initialized, Role: FOLLOWER
I20260619 06:53:35.142944 30554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } }
I20260619 06:53:35.143112 30554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:35.143200 30554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:35.143316 30554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:35.144438 30554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } }
I20260619 06:53:35.144829 30554 leader_election.cc:302] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19f50a85faf44438b02769f4abedd21; no voters:
I20260619 06:53:35.145174 30554 leader_election.cc:288] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:35.145272 30559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:35.145445 30559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [term 1 LEADER]: Becoming Leader. State: Replica: f19f50a85faf44438b02769f4abedd21, State: Running, Role: LEADER
I20260619 06:53:35.145741 30559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } }
I20260619 06:53:35.146463 30554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:35.146981 30559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19f50a85faf44438b02769f4abedd21. Latest consensus state: current_term: 1 leader_uuid: "f19f50a85faf44438b02769f4abedd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } } }
I20260619 06:53:35.146968 30561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19f50a85faf44438b02769f4abedd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19f50a85faf44438b02769f4abedd21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41421 } } }
I20260619 06:53:35.147141 30559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:35.147178 30561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21 [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:35.148955 30564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:35.150383 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.134186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.22.113.62:41421)
0619 06:53:35.134820 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.134849 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.134918 (+ 69us) server_negotiation.cc:407] Connection header received
0619 06:53:35.135926 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.135951 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.136382 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.136794 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.137738 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.139302 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.140371 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.141090 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.141267 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.144331 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.144340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.144359 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.144398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.146604 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.147350 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.147357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.147364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.147537 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.147882 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.147887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.147889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.148508 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.148750 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.149100 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.149423 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":168,"threads_started":1}
I20260619 06:53:35.152637 30564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:35.159329 30564 catalog_manager.cc:1347] Generated new cluster ID: 7dac8d96096d4cc49a29372006ce0a0c
I20260619 06:53:35.159404 30564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:35.165237 30564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:35.166709 30564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:35.179075 30564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Generated new TSK 0
I20260619 06:53:35.180079 30564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
I20260619 06:53:35.215190 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41421
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:35.332439 30580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:35.332744 30580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:35.332823 30580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:35.336882 30580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:35.336968 30580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:35.337046 30580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:35.337131 30580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:35.342316 30580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:35.343483 30580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:35.344525 30580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:53:35.356546 30580 server_base.cc:1034] running on GCE node
W20260619 06:53:35.356446 30585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.356462 30586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.356464 30588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:35.357297 30580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:35.358023 30580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:35.359217 30580 hybrid_clock.cc:648] HybridClock initialized: now 1781852015359175 us; error 59 us; skew 500 ppm
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:35.362385 30580 init.cc:377] Logged in from keytab as oryx/127.22.113.1@KRBTEST.COM (short username oryx)
I20260619 06:53:35.363531 30580 webserver.cc:460] Webserver started at http://127.22.113.1:45059/ using document root <none> and password file <none>
I20260619 06:53:35.364365 30580 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:35.364465 30580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:35.364733 30580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:35.367079 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "03e96693820643c2bf8289f1ab417f20"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7401a54f9c4f745d0b41221c6ff2b2ef"
server_key_iv: "d13ef4902be065a0c34a94be010aadcc"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.367714 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "03e96693820643c2bf8289f1ab417f20"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7401a54f9c4f745d0b41221c6ff2b2ef"
server_key_iv: "d13ef4902be065a0c34a94be010aadcc"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.371966 30580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:35.374737 30595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.376368 30580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:35.376507 30580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "03e96693820643c2bf8289f1ab417f20"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "7401a54f9c4f745d0b41221c6ff2b2ef"
server_key_iv: "d13ef4902be065a0c34a94be010aadcc"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.376626 30580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:35.391422 30580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:35.394672 30580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:35.394850 30580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:35.396497 30580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:35.396576 30580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.396646 30580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:35.396689 30580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:35.409253 30580 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:45675
I20260619 06:53:35.409341 30710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:45675 every 8 connection(s)
I20260619 06:53:35.410514 30580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:35.413275 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.397661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:41433 (local address 127.22.113.62:41421)
0619 06:53:35.397832 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.397838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.398734 (+ 896us) server_negotiation.cc:407] Connection header received
0619 06:53:35.399958 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.399965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.400054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.400192 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.401791 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.402737 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.404379 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.404643 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.404783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.409115 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.409119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.409121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.409152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.410891 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.411584 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.411589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.411590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.411669 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.412039 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.412041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.412042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.412219 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.412311 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.412912 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.413027 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:53:35.413375 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30580
I20260619 06:53:35.413497 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:35.413818 22980 external_mini_cluster.cc:1428] Setting key 5e2b8f65b6655e77216b083645d898c5
I20260619 06:53:35.414157 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.398061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.1:41433)
0619 06:53:35.398554 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.398606 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.399699 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.400389 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.400397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.400946 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.401608 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.401628 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.402884 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.402888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.404218 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.404229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.404863 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.404869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.405164 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.406834 (+ 1670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.406861 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.408927 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.411040 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.411049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.411068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.411447 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.411775 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.411778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.411780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.411931 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.412352 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.412357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.412637 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.412807 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.413107 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":150,"threads_started":1}
I20260619 06:53:35.415985 30711 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41421
I20260619 06:53:35.416163 30711 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:35.416713 30711 heartbeater.cc:511] Master 127.22.113.62:41421 requested a full tablet report, sending...
I20260619 06:53:35.419171 30508 ts_manager.cc:194] Registered new tserver with Master: 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675)
I20260619 06:53:35.420833 30508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:41433
WARNING: no policy specified for oryx/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:35.431421 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.421391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:45373 (local address 127.22.113.62:41421)
0619 06:53:35.421562 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.421567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.423658 (+ 2091us) server_negotiation.cc:407] Connection header received
0619 06:53:35.423783 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.423785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.423854 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.423939 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.424566 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.425163 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.426201 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.426433 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.426534 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.427836 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.427840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.427843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.427870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.429599 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.430098 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.430103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.430105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.430190 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.430493 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.430496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.430498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.430684 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.430794 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.431064 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.431186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:53:35.431429 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.421340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.1:45373)
0619 06:53:35.423514 (+ 2174us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.423534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.423663 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.424029 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.424033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.424286 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.424437 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.424445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.425295 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.425298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.425962 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.425971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.426555 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.426558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.426628 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.427299 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.427313 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.427628 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.429742 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.429746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.429749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.429987 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.430291 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.430294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.430296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.430394 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.430787 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.430790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.430891 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.430997 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.431180 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
I20260619 06:53:35.446604 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41421
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:35.555852 30720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:35.556160 30720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:35.556231 30720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:35.560716 30720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:35.560799 30720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:35.560861 30720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:35.560945 30720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:35.566179 30720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:35.567159 30720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:35.568148 30720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:35.579301 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.579216 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.579291 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:35.579576 30720 server_base.cc:1034] running on GCE node
I20260619 06:53:35.579990 30720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:35.580646 30720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:35.581818 30720 hybrid_clock.cc:648] HybridClock initialized: now 1781852015581761 us; error 68 us; skew 500 ppm
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:35.584782 30720 init.cc:377] Logged in from keytab as oryx/127.22.113.2@KRBTEST.COM (short username oryx)
I20260619 06:53:35.585723 30720 webserver.cc:460] Webserver started at http://127.22.113.2:44947/ using document root <none> and password file <none>
I20260619 06:53:35.586304 30720 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:35.586393 30720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:35.586627 30720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:35.588765 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "c53a3a2a9033033d07fbf86844ad6d25"
server_key_iv: "816bf7d14fd42c9cb91396f712ea7105"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.589323 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "c53a3a2a9033033d07fbf86844ad6d25"
server_key_iv: "816bf7d14fd42c9cb91396f712ea7105"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.593242 30720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:35.595866 30735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.597227 30720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:35.597369 30720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "c53a3a2a9033033d07fbf86844ad6d25"
server_key_iv: "816bf7d14fd42c9cb91396f712ea7105"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.597481 30720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:35.608459 30720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:35.611707 30720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:35.611907 30720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:35.613569 30720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:35.613639 30720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.613708 30720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:35.613758 30720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:35.625176 30720 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41215
I20260619 06:53:35.625357 30850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41215 every 8 connection(s)
I20260619 06:53:35.626343 30720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:35.629446 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.614824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57231 (local address 127.22.113.62:41421)
0619 06:53:35.614989 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.614993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.616271 (+ 1278us) server_negotiation.cc:407] Connection header received
0619 06:53:35.617922 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.617927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.618012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.618163 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.619819 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.620421 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.621699 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.621941 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.622057 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.625490 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.625494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.625498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.625544 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.627306 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.627859 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.627862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.627864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.627947 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.628295 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.628299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.628301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.628510 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.628643 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.629137 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.629264 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:53:35.630211 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.615453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.2:57231)
0619 06:53:35.616097 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.616161 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.617594 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.618386 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.618395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.618948 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.619655 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.619672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.620557 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.620561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.621570 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.621580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.622106 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.622111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.622285 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.623013 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.623033 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.625230 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.627458 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.627463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.627477 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.627744 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.628053 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.628056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.628058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.628194 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.628638 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.628644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.628894 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.629082 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.629406 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":138,"threads_started":1}
I20260619 06:53:35.631728 30851 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41421
I20260619 06:53:35.631948 30851 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:35.632707 30851 heartbeater.cc:511] Master 127.22.113.62:41421 requested a full tablet report, sending...
I20260619 06:53:35.633384 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30720
I20260619 06:53:35.633528 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:35.633867 22980 external_mini_cluster.cc:1428] Setting key ef101000ba1929172dd1d2426e87470f
I20260619 06:53:35.634078 30507 ts_manager.cc:194] Registered new tserver with Master: 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215)
I20260619 06:53:35.635133 30507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:57231
I20260619 06:53:35.643597 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.634496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.2:55955)
0619 06:53:35.634947 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.634969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.635295 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.635798 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.635803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.636088 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.636212 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.636219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.637112 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.637116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.637726 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.637735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.638429 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.638433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.638547 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.639105 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.639117 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.639553 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.641707 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.641709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.641710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.641905 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.642238 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.642240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.642242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.642319 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.642942 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.642945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.643041 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.643140 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.643441 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
I20260619 06:53:35.643867 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.634347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:55955 (local address 127.22.113.62:41421)
0619 06:53:35.634468 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.634473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.635079 (+ 606us) server_negotiation.cc:407] Connection header received
0619 06:53:35.635435 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.635535 (+ 100us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.635592 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.635649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.636352 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.636967 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.637874 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.638255 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.638386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.639718 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.639725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.639731 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.639764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.641571 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.642011 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.642018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.642023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.642121 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.642428 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.642433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.642436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.642809 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.642944 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.643367 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.643559 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for oryx/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
I20260619 06:53:35.667786 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:41421
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:35.796831 30860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:35.797171 30860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:35.797257 30860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:35.801569 30860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:35.801661 30860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:35.801729 30860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:53:35.801821 30860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:35.807351 30860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.113.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:35.808750 30860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:35.809973 30860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:35.821028 30868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.821041 30865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:35.821058 30866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:35.821496 30860 server_base.cc:1034] running on GCE node
I20260619 06:53:35.822016 30860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:35.822772 30860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:35.823973 30860 hybrid_clock.cc:648] HybridClock initialized: now 1781852015823948 us; error 39 us; skew 500 ppm
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:35.827085 30860 init.cc:377] Logged in from keytab as oryx/127.22.113.3@KRBTEST.COM (short username oryx)
I20260619 06:53:35.828058 30860 webserver.cc:460] Webserver started at http://127.22.113.3:33223/ using document root <none> and password file <none>
I20260619 06:53:35.828598 30860 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:35.828660 30860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:35.828821 30860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:35.830824 30860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "e13c15c9c8ee4c99843b284686efe607"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "b2814371625e419d142b67d7accd382a"
server_key_iv: "69b4c8a5dfbffb9f776eeafd5e2c3a56"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.831419 30860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "e13c15c9c8ee4c99843b284686efe607"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "b2814371625e419d142b67d7accd382a"
server_key_iv: "69b4c8a5dfbffb9f776eeafd5e2c3a56"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.835865 30860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:35.838529 30875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.840018 30860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:35.840158 30860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "e13c15c9c8ee4c99843b284686efe607"
format_stamp: "Formatted at 2026-06-19 06:53:35 on dist-test-slave-g8zg"
server_key: "b2814371625e419d142b67d7accd382a"
server_key_iv: "69b4c8a5dfbffb9f776eeafd5e2c3a56"
server_key_version: "encryptionkey@0"
I20260619 06:53:35.840271 30860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:35.862428 30860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:35.865752 30860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:35.865960 30860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:35.867583 30860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:35.867657 30860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:35.867720 30860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:35.867761 30860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:35.879741 30860 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38077
I20260619 06:53:35.879823 30990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38077 every 8 connection(s)
I20260619 06:53:35.880834 30860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:35.883929 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.868893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:35113 (local address 127.22.113.62:41421)
0619 06:53:35.869078 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.869083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.870112 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 06:53:35.871529 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.871533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.871599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.871689 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.873462 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.874262 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.876308 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.876569 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.876656 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.880156 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.880160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.880163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.880190 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.882062 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.882543 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.882546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.882548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.882628 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.882883 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.882885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.882886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.883053 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.883138 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.883696 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.883802 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:53:35.884878 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.869378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.3:35113)
0619 06:53:35.869912 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.869968 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.871262 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.871894 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.871902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.872437 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.873246 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.873273 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.874445 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.874453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.876118 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.876135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.876758 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.876762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.876976 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.877963 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.877986 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.879987 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.882173 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.882177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.882187 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.882452 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.882692 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.882694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.882696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.882820 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.883142 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.883147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.883459 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.883621 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.883914 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260619 06:53:35.885134 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 30860
I20260619 06:53:35.885289 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:35.885681 22980 external_mini_cluster.cc:1428] Setting key 98ab695b48746bb73e014dfd86e71200
I20260619 06:53:35.886157 30991 heartbeater.cc:347] Connected to a master server at 127.22.113.62:41421
I20260619 06:53:35.886427 30991 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:35.887147 30991 heartbeater.cc:511] Master 127.22.113.62:41421 requested a full tablet report, sending...
I20260619 06:53:35.888414 30507 ts_manager.cc:194] Registered new tserver with Master: e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077)
I20260619 06:53:35.889297 30507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:35113
I20260619 06:53:35.896373 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.888971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.22.113.3:39177)
0619 06:53:35.889684 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.889700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.889808 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.890076 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.890078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.890266 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:35.890368 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.890373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.891422 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.891425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.892033 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.892041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.892476 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.892478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.892543 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.893008 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:35.893019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:35.893328 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:35.895019 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.895021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.895023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.895192 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.895526 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:35.895528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:35.895529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:35.895601 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:35.895972 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:35.895974 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:35.896045 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.896133 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.896232 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":149,"threads_started":1}
I20260619 06:53:35.896381 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.889082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:39177 (local address 127.22.113.62:41421)
0619 06:53:35.889226 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.889231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.889783 (+ 552us) server_negotiation.cc:407] Connection header received
0619 06:53:35.889912 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.889915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.889966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.890064 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.890499 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.891311 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.892144 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.892368 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.892455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.893454 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.893457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.893459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.893480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.894924 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.895367 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.895370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.895371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.895440 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.895715 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.895717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.895718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.895887 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.895966 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.896167 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.896257 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:53:35.900028 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:53:35.925997 30557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:35.918996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.22.113.62:41421)
0619 06:53:35.919157 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.919160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.919175 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:53:35.919219 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.919222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.919335 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.919414 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.920195 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.920794 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.921690 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.921900 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.921994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.925825 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60738: BlockingRecv error: recv got EOF from 127.0.0.1:60738 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:53:35 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:35.936168 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.927556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.22.113.62:41421)
0619 06:53:35.927729 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.927734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.927868 (+ 134us) server_negotiation.cc:407] Connection header received
0619 06:53:35.928022 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.928025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.928066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.928174 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:35.928775 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.929279 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.930064 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.930297 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.930400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.933006 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:35.933010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:35.933013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:35.933042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:35.934509 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.935005 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.935009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.935011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.935095 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:35.935424 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:35.935426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:35.935427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:35.935616 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:35.935725 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.935934 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.936031 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:53:35.938982 30507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:35.942063 30507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:35.958343 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.952677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.0.0.1:58068)
0619 06:53:35.953939 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.953948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.954060 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.954627 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.954629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.954643 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:35.954698 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.954702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955910 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.955914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.957003 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.957010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.957771 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.957775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.957885 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.957920 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.958061 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.958189 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260619 06:53:35.958343 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.952894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.22.113.1:45675)
0619 06:53:35.953549 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.953556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.954057 (+ 501us) server_negotiation.cc:407] Connection header received
0619 06:53:35.954166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.954171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.954361 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.954496 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:35.954807 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955778 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.957115 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.957650 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.957773 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.957921 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.958090 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.958185 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":97,"threads_started":1}
I20260619 06:53:35.958753 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.951846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38077 (local address 127.0.0.1:59868)
0619 06:53:35.952566 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.952614 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.952743 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.953581 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.953589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.953615 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:35.953761 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.953767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955287 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.955291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.956685 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.956695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.958056 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.958060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.958306 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.958393 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.958473 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.958585 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":123,"thread_start_us":138,"threads_started":1}
I20260619 06:53:35.958797 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.952161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41215 (local address 127.0.0.1:36382)
0619 06:53:35.953651 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:35.953664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:35.953759 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:35.954277 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:35.954281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:35.954296 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:35.954384 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.954390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955626 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.955633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.956746 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:35.956782 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.958056 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.958060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:35.958309 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.958393 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 06:53:35.958616 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:53:35.958678 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"mutex_wait_us":116,"thread_start_us":139,"threads_started":1}
I20260619 06:53:35.959446 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.952766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.22.113.2:41215)
0619 06:53:35.953272 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.953284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.953769 (+ 485us) server_negotiation.cc:407] Connection header received
0619 06:53:35.953889 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.953894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.954121 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.954333 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:35.954467 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955504 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.956919 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.957695 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.957807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.957988 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.959146 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.959287 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":137,"threads_started":1}
I20260619 06:53:35.959661 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:35.952072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.22.113.3:38077)
0619 06:53:35.952439 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:35.952453 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:35.952894 (+ 441us) server_negotiation.cc:407] Connection header received
0619 06:53:35.952974 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:35.952979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:35.953192 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:35.953351 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:35.953887 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.955128 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.956830 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:35.957695 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:35.957807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:35.957995 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:35.959372 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0619 06:53:35.959518 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260619 06:53:35.961220 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 (DEFAULT_TABLE table=test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:35.962116 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbc0e2075a94b369d189c6de1d08423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:35.962096 30785 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 (DEFAULT_TABLE table=test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:35.963171 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbc0e2075a94b369d189c6de1d08423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:35.965001 30925 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 (DEFAULT_TABLE table=test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:35.965986 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbc0e2075a94b369d189c6de1d08423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:35.970221 31020 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Bootstrap starting.
I20260619 06:53:35.970664 31019 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Bootstrap starting.
I20260619 06:53:35.972971 31019 tablet_bootstrap.cc:654] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:35.973440 31020 tablet_bootstrap.cc:654] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:35.973667 31021 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Bootstrap starting.
I20260619 06:53:35.973927 31019 log.cc:824] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:35.974632 31020 log.cc:824] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:35.975961 31019 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: No bootstrap required, opened a new log
I20260619 06:53:35.976122 31021 tablet_bootstrap.cc:654] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:35.976181 31019 ts_tablet_manager.cc:1397] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 06:53:35.976981 31021 log.cc:824] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:35.977051 31020 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: No bootstrap required, opened a new log
I20260619 06:53:35.977233 31020 ts_tablet_manager.cc:1397] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:53:35.980161 31019 raft_consensus.cc:348] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.980430 31019 raft_consensus.cc:374] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:35.980508 31019 raft_consensus.cc:729] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Initialized, Role: FOLLOWER
I20260619 06:53:35.980778 31020 raft_consensus.cc:348] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.981045 31020 raft_consensus.cc:374] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:35.980968 31019 consensus_queue.cc:260] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.981132 31020 raft_consensus.cc:729] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:35.981700 30711 heartbeater.cc:503] Master 127.22.113.62:41421 was elected leader, sending a full tablet report...
I20260619 06:53:35.981642 31020 consensus_queue.cc:260] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.981971 31019 ts_tablet_manager.cc:1428] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 06:53:35.982460 30851 heartbeater.cc:503] Master 127.22.113.62:41421 was elected leader, sending a full tablet report...
I20260619 06:53:35.982673 31020 ts_tablet_manager.cc:1428] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:53:35.984814 31021 tablet_bootstrap.cc:492] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: No bootstrap required, opened a new log
I20260619 06:53:35.985050 31021 ts_tablet_manager.cc:1397] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260619 06:53:35.989643 31021 raft_consensus.cc:348] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.989884 31021 raft_consensus.cc:374] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:35.989920 31021 raft_consensus.cc:729] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Initialized, Role: FOLLOWER
I20260619 06:53:35.990309 31021 consensus_queue.cc:260] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:35.991025 30991 heartbeater.cc:503] Master 127.22.113.62:41421 was elected leader, sending a full tablet report...
I20260619 06:53:35.991376 31021 ts_tablet_manager.cc:1428] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:53:36.111891 31027 raft_consensus.cc:482] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:36.112151 31027 raft_consensus.cc:504] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:36.113376 31027 leader_election.cc:288] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215), 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675)
I20260619 06:53:36.118741 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.113949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:42843 (local address 127.22.113.2:41215)
0619 06:53:36.114091 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.114097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.114459 (+ 362us) server_negotiation.cc:407] Connection header received
0619 06:53:36.114540 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.114543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.114601 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.114700 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:36.115109 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.115912 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.117720 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.118150 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.118290 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.118350 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.118543 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.118620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:53:36.118763 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.113874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41215 (local address 127.22.113.3:42843)
0619 06:53:36.114326 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.114344 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.114454 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.114787 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.114792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.114814 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:36.114959 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.114966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.116033 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.116038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.117521 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.117531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.118283 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.118286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.118391 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.118420 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.118526 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.118600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":33,"thread_start_us":208,"threads_started":1}
I20260619 06:53:36.118971 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.113983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:58021 (local address 127.22.113.1:45675)
0619 06:53:36.114098 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.114104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.114121 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:53:36.114245 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.114248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.114302 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.114412 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:36.114827 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.115562 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.117690 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.118212 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.118463 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.118524 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.118727 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.118817 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:53:36.118992 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.113718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.22.113.3:58021)
0619 06:53:36.113953 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.113973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.114105 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.114480 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.114485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.114510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:36.114686 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.114696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.115686 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.115691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.117521 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.117531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.118452 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.118457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.118577 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.118607 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.118700 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.118822 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":50}
I20260619 06:53:36.119745 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbc0e2075a94b369d189c6de1d08423" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" is_pre_election: true
I20260619 06:53:36.119720 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbc0e2075a94b369d189c6de1d08423" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20" is_pre_election: true
I20260619 06:53:36.120042 30805 raft_consensus.cc:2454] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:36.120042 30665 raft_consensus.cc:2454] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:36.120589 30876 leader_election.cc:302] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:36.120839 31027 raft_consensus.cc:2781] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:36.120921 31027 raft_consensus.cc:482] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:36.120963 31027 raft_consensus.cc:3037] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.122066 31027 raft_consensus.cc:504] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
I20260619 06:53:36.122453 31027 leader_election.cc:288] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Requested vote from peers 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215), 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675)
I20260619 06:53:36.122803 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbc0e2075a94b369d189c6de1d08423" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20"
I20260619 06:53:36.122800 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbc0e2075a94b369d189c6de1d08423" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
I20260619 06:53:36.122941 30665 raft_consensus.cc:3037] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.122969 30805 raft_consensus.cc:3037] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.123996 30665 raft_consensus.cc:2454] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:36.124303 30877 leader_election.cc:302] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:36.124486 31027 raft_consensus.cc:2781] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:36.124567 30805 raft_consensus.cc:2454] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:36.124725 31027 raft_consensus.cc:686] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Becoming Leader. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Running, Role: LEADER
I20260619 06:53:36.125064 31027 consensus_queue.cc:237] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } }
W20260619 06:53:36.127761 30852 tablet.cc:2369] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:36.128528 30507 catalog_manager.cc:5495] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13c15c9c8ee4c99843b284686efe607 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "e13c15c9c8ee4c99843b284686efe607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } health_report { overall_health: UNKNOWN } } }
W20260619 06:53:36.132613 30992 tablet.cc:2369] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:36.154140 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.150523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.22.113.3:38077)
0619 06:53:36.150714 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.150718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.150740 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:53:36.150807 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.150809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.150860 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.150969 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:36.151219 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.151790 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.152659 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.152883 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.152989 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.153305 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.153749 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.153875 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.153938 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.153991 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 06:53:36.160887 30805 raft_consensus.cc:1261] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Refusing update from remote peer e13c15c9c8ee4c99843b284686efe607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:36.160915 30665 raft_consensus.cc:1261] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Refusing update from remote peer e13c15c9c8ee4c99843b284686efe607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:36.161717 31029 consensus_queue.cc:1035] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:36.161909 31027 consensus_queue.cc:1035] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:53:36.162125 30712 tablet.cc:2369] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:53:36.171105 31034 mvcc.cc:205] Tried to move back new op lower bound from 7298465858186108928 to 7298465858051067904. Current Snapshot: MvccSnapshot[applied={T|T < 7298465858186108928}]
I20260619 06:53:36.178821 30508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:36.178975 30508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:36.182051 30508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Sending DeleteTablet for 3 replicas of tablet 8dbc0e2075a94b369d189c6de1d08423
I20260619 06:53:36.182976 30785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:36382
I20260619 06:53:36.182978 30925 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260619 06:53:36.183363 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbc0e2075a94b369d189c6de1d08423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:58068
I20260619 06:53:36.183473 31042 tablet_replica.cc:330] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: stopping tablet replica
I20260619 06:53:36.183799 31042 raft_consensus.cc:2229] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:36.184013 31043 tablet_replica.cc:330] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: stopping tablet replica
I20260619 06:53:36.184134 31042 raft_consensus.cc:2258] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:36.184370 31043 raft_consensus.cc:2229] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:36.184670 30508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:53:36.184816 31043 raft_consensus.cc:2258] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:53:36.185098 30508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:36.185292 31044 tablet_replica.cc:330] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: stopping tablet replica
I20260619 06:53:36.185588 31044 raft_consensus.cc:2229] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:36.185698 31042 ts_tablet_manager.cc:1905] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.185927 31044 raft_consensus.cc:2258] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:36.188146 31043 ts_tablet_manager.cc:1905] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.189153 31042 ts_tablet_manager.cc:1918] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:36.189270 31042 log.cc:1196] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/8dbc0e2075a94b369d189c6de1d08423
I20260619 06:53:36.189685 31042 ts_tablet_manager.cc:1939] T 8dbc0e2075a94b369d189c6de1d08423 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting consensus metadata
I20260619 06:53:36.190117 31044 ts_tablet_manager.cc:1905] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.191088 30492 catalog_manager.cc:4841] TS 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215): tablet 8dbc0e2075a94b369d189c6de1d08423 (table test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]) successfully deleted
I20260619 06:53:36.190790 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 (DEFAULT_TABLE table=test-table [id=fdd24999ffa54c15b63af127c0396fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.191387 30785 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 (DEFAULT_TABLE table=test-table [id=fdd24999ffa54c15b63af127c0396fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.191494 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e51af8d914ee9ab3452452eb2fa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.191627 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e51af8d914ee9ab3452452eb2fa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.193774 31020 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Bootstrap starting.
I20260619 06:53:36.194399 31019 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Bootstrap starting.
I20260619 06:53:36.195053 31020 tablet_bootstrap.cc:654] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.195520 31019 tablet_bootstrap.cc:654] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.195566 30925 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 (DEFAULT_TABLE table=test-table [id=fdd24999ffa54c15b63af127c0396fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.196823 31044 ts_tablet_manager.cc:1918] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:36.197140 31043 ts_tablet_manager.cc:1918] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:36.197266 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7e51af8d914ee9ab3452452eb2fa40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.197353 31020 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: No bootstrap required, opened a new log
I20260619 06:53:36.197386 31043 log.cc:1196] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/8dbc0e2075a94b369d189c6de1d08423
I20260619 06:53:36.197449 31020 ts_tablet_manager.cc:1397] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:36.197918 31043 ts_tablet_manager.cc:1939] T 8dbc0e2075a94b369d189c6de1d08423 P e13c15c9c8ee4c99843b284686efe607: Deleting consensus metadata
I20260619 06:53:36.197942 31020 raft_consensus.cc:348] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.198081 31020 raft_consensus.cc:374] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.198120 31020 raft_consensus.cc:729] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.198240 31020 consensus_queue.cc:260] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.198503 31020 ts_tablet_manager.cc:1428] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:36.199882 31044 log.cc:1196] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/8dbc0e2075a94b369d189c6de1d08423
I20260619 06:53:36.199919 30494 catalog_manager.cc:4841] TS e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077): tablet 8dbc0e2075a94b369d189c6de1d08423 (table test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]) successfully deleted
I20260619 06:53:36.200543 31021 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Bootstrap starting.
I20260619 06:53:36.201081 31044 ts_tablet_manager.cc:1939] T 8dbc0e2075a94b369d189c6de1d08423 P 03e96693820643c2bf8289f1ab417f20: Deleting consensus metadata
I20260619 06:53:36.201651 31021 tablet_bootstrap.cc:654] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.202890 30493 catalog_manager.cc:4841] TS 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675): tablet 8dbc0e2075a94b369d189c6de1d08423 (table test-table [id=b84c98b34e2946e2b738d3c3b3e5a811]) successfully deleted
I20260619 06:53:36.196992 31019 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: No bootstrap required, opened a new log
I20260619 06:53:36.203409 31019 ts_tablet_manager.cc:1397] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 06:53:36.203641 31021 tablet_bootstrap.cc:492] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: No bootstrap required, opened a new log
I20260619 06:53:36.203729 31021 ts_tablet_manager.cc:1397] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:36.203820 31019 raft_consensus.cc:348] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.203907 31019 raft_consensus.cc:374] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.203928 31019 raft_consensus.cc:729] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.204066 31019 consensus_queue.cc:260] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.204339 31019 ts_tablet_manager.cc:1428] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:36.204281 31021 raft_consensus.cc:348] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.204409 31021 raft_consensus.cc:374] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.204447 31021 raft_consensus.cc:729] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.204591 31021 consensus_queue.cc:260] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.205185 31021 ts_tablet_manager.cc:1428] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:36.215947 31038 raft_consensus.cc:482] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:36.216061 31038 raft_consensus.cc:504] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.216564 31038 leader_election.cc:288] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215)
I20260619 06:53:36.216964 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e51af8d914ee9ab3452452eb2fa40" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20" is_pre_election: true
I20260619 06:53:36.217096 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e51af8d914ee9ab3452452eb2fa40" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" is_pre_election: true
I20260619 06:53:36.217203 30665 raft_consensus.cc:2454] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:36.217267 30805 raft_consensus.cc:2454] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:36.217592 30876 leader_election.cc:302] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:36.217734 31038 raft_consensus.cc:2781] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:36.217782 31038 raft_consensus.cc:482] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:36.217836 31038 raft_consensus.cc:3037] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.218788 31038 raft_consensus.cc:504] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.219146 31038 leader_election.cc:288] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Requested vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215)
I20260619 06:53:36.219544 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e51af8d914ee9ab3452452eb2fa40" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
I20260619 06:53:36.219581 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7e51af8d914ee9ab3452452eb2fa40" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20"
I20260619 06:53:36.219707 30805 raft_consensus.cc:3037] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.219738 30665 raft_consensus.cc:3037] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.220642 30805 raft_consensus.cc:2454] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:36.220971 30876 leader_election.cc:302] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:36.221210 30665 raft_consensus.cc:2454] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:36.221306 31038 raft_consensus.cc:2781] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:36.221395 31038 raft_consensus.cc:686] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Becoming Leader. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Running, Role: LEADER
I20260619 06:53:36.221599 31038 consensus_queue.cc:237] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.223534 30507 catalog_manager.cc:5495] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13c15c9c8ee4c99843b284686efe607 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "e13c15c9c8ee4c99843b284686efe607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } health_report { overall_health: HEALTHY } } }
Jun 19 06:53:36 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852015, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.62@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:36.267204 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.257061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:41421 (local address 127.0.0.1:60754)
0619 06:53:36.257341 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.257364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.257453 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.257894 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.257897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.258292 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:36.258405 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.258412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.259607 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.259610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.260196 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.260204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.260812 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.260819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.260909 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.261595 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:36.261607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:36.263448 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:36.265655 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:36.265663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:36.265667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:36.265906 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:36.266206 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:36.266208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:36.266209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:36.266285 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:36.266733 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:36.266738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:36.266841 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.266939 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.267061 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260619 06:53:36.267403 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.257208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.22.113.62:41421)
0619 06:53:36.257399 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.257404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.257431 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:53:36.257579 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.257583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.257652 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.257795 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:36.258549 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.259453 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.260322 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.260677 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.260787 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.263626 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:36.263631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:36.263634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:36.263663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:36.265529 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:36.266014 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:36.266018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:36.266020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:36.266109 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:36.266377 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:36.266381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:36.266383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:36.266615 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:36.266720 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.267025 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.267152 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:53:36.271350 30507 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:53:36.272120 30507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:36.277442 30645 tablet_service.cc:1467] Processing CreateTablet for tablet ef217df3dafa46c594cb5406db3ed053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1190e38a4e04a1bae7f2e65f3e2cf22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:36.277717 30925 tablet_service.cc:1467] Processing CreateTablet for tablet ef217df3dafa46c594cb5406db3ed053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1190e38a4e04a1bae7f2e65f3e2cf22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:36.277854 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef217df3dafa46c594cb5406db3ed053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.278039 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef217df3dafa46c594cb5406db3ed053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.280772 31019 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: Bootstrap starting.
I20260619 06:53:36.280814 31021 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: Bootstrap starting.
I20260619 06:53:36.281258 30785 tablet_service.cc:1467] Processing CreateTablet for tablet ef217df3dafa46c594cb5406db3ed053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1190e38a4e04a1bae7f2e65f3e2cf22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:53:36.281594 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef217df3dafa46c594cb5406db3ed053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.281728 31019 tablet_bootstrap.cc:654] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.282094 31021 tablet_bootstrap.cc:654] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.283478 31019 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: No bootstrap required, opened a new log
I20260619 06:53:36.283579 31019 ts_tablet_manager.cc:1397] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:36.284036 31019 raft_consensus.cc:348] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.284163 31019 raft_consensus.cc:374] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.284198 31019 raft_consensus.cc:729] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.284324 31019 consensus_queue.cc:260] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.284626 31019 ts_tablet_manager.cc:1428] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:36.284835 31025 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.284991 31025 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.285166 31020 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: Bootstrap starting.
I20260619 06:53:36.286008 31020 tablet_bootstrap.cc:654] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.287458 31020 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: No bootstrap required, opened a new log
I20260619 06:53:36.287540 31020 ts_tablet_manager.cc:1397] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:53:36.288026 31020 raft_consensus.cc:348] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.288116 31021 tablet_bootstrap.cc:492] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: No bootstrap required, opened a new log
I20260619 06:53:36.288158 31020 raft_consensus.cc:374] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.288200 31020 raft_consensus.cc:729] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.288211 31021 ts_tablet_manager.cc:1397] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260619 06:53:36.288322 31020 consensus_queue.cc:260] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.288621 31020 ts_tablet_manager.cc:1428] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:53:36.288559 31026 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.288699 31026 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.288707 31021 raft_consensus.cc:348] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.288834 31021 raft_consensus.cc:374] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.288872 31021 raft_consensus.cc:729] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.289024 31021 consensus_queue.cc:260] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.289289 31021 ts_tablet_manager.cc:1428] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:36.289292 31029 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.289664 31029 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.569396 31025 raft_consensus.cc:482] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:36.569607 31025 raft_consensus.cc:504] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.570789 31025 leader_election.cc:288] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215), e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077)
I20260619 06:53:36.576799 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.571184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38077 (local address 127.22.113.1:43057)
0619 06:53:36.571626 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.571642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.571778 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.572216 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.572222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.572241 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:36.572380 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.572387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.573502 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.573506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.575180 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.575195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.576176 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.576180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.576272 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.576368 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.576455 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.576598 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":195,"thread_start_us":148,"threads_started":1}
I20260619 06:53:36.576810 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:41817 (local address 127.22.113.2:41215)
0619 06:53:36.571481 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.571487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.571844 (+ 357us) server_negotiation.cc:407] Connection header received
0619 06:53:36.572106 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.572109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.572208 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.572318 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:36.572517 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.573287 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.575323 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.575969 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.576109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.576182 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.576538 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.576640 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:53:36.576810 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:43057 (local address 127.22.113.3:38077)
0619 06:53:36.571631 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.571636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.571750 (+ 114us) server_negotiation.cc:407] Connection header received
0619 06:53:36.571948 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.571952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.572012 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.572088 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:36.572516 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.573321 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.575399 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.576061 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.576184 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.576242 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.576542 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.576640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260619 06:53:36.576799 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.571195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41215 (local address 127.22.113.1:41817)
0619 06:53:36.571662 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.571678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.571859 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.572337 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.572340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.572354 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:36.572434 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.572443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.573416 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.573423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.575158 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.575169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.576107 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.576110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.576270 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.576367 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.576455 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.576632 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":288,"thread_start_us":187,"threads_started":1}
I20260619 06:53:36.577718 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef217df3dafa46c594cb5406db3ed053" candidate_uuid: "03e96693820643c2bf8289f1ab417f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13c15c9c8ee4c99843b284686efe607" is_pre_election: true
I20260619 06:53:36.577904 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef217df3dafa46c594cb5406db3ed053" candidate_uuid: "03e96693820643c2bf8289f1ab417f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" is_pre_election: true
I20260619 06:53:36.577983 30945 raft_consensus.cc:2454] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e96693820643c2bf8289f1ab417f20 in term 0.
I20260619 06:53:36.578100 30805 raft_consensus.cc:2454] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e96693820643c2bf8289f1ab417f20 in term 0.
I20260619 06:53:36.578625 30598 leader_election.cc:302] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:36.578971 31025 raft_consensus.cc:2781] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:36.579044 31025 raft_consensus.cc:482] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:36.579123 31025 raft_consensus.cc:3037] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.580541 31025 raft_consensus.cc:504] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.580921 31025 leader_election.cc:288] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [CANDIDATE]: Term 1 election: Requested vote from peers 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215), e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077)
I20260619 06:53:36.581337 30805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef217df3dafa46c594cb5406db3ed053" candidate_uuid: "03e96693820643c2bf8289f1ab417f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
I20260619 06:53:36.581466 30805 raft_consensus.cc:3037] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.582334 30805 raft_consensus.cc:2454] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e96693820643c2bf8289f1ab417f20 in term 1.
I20260619 06:53:36.582608 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef217df3dafa46c594cb5406db3ed053" candidate_uuid: "03e96693820643c2bf8289f1ab417f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13c15c9c8ee4c99843b284686efe607"
I20260619 06:53:36.582684 30596 leader_election.cc:302] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec; no voters:
I20260619 06:53:36.582737 30945 raft_consensus.cc:3037] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:36.582939 31025 raft_consensus.cc:2781] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:36.583214 31025 raft_consensus.cc:686] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [term 1 LEADER]: Becoming Leader. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Running, Role: LEADER
I20260619 06:53:36.583539 31025 consensus_queue.cc:237] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.583881 30945 raft_consensus.cc:2454] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e96693820643c2bf8289f1ab417f20 in term 1.
I20260619 06:53:36.585176 31060 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: TxnStatusTablet state changed. Reason: New leader 03e96693820643c2bf8289f1ab417f20. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.585299 31060 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:36.585602 31062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:53:36.585775 31062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:53:36.586256 30507 catalog_manager.cc:5495] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e96693820643c2bf8289f1ab417f20 (127.22.113.1). New cstate: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:36.599203 31029 consensus_queue.cc:1035] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:36.604380 31038 consensus_queue.cc:1035] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:36.660828 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.656120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.22.113.1:45675)
0619 06:53:36.656440 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.656444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.656574 (+ 130us) server_negotiation.cc:407] Connection header received
0619 06:53:36.656815 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.656817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.656891 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.656997 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:36.657297 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.657881 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.659143 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.659471 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.659622 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.659986 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.660413 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.660546 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.660631 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.660693 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260619 06:53:36.660909 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.655976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.0.0.1:58084)
0619 06:53:36.656475 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.656494 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.656650 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.657035 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.657038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.657047 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:36.657189 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.657195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.657992 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.657996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.659001 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.659012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.659644 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.659648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.659761 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.659827 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.660515 (+ 688us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.660531 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.660620 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.660721 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356}
I20260619 06:53:36.664395 30805 raft_consensus.cc:1261] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Refusing update from remote peer 03e96693820643c2bf8289f1ab417f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:36.664716 30945 raft_consensus.cc:1261] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Refusing update from remote peer 03e96693820643c2bf8289f1ab417f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:36.664865 31025 consensus_queue.cc:1035] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:36.665343 31025 consensus_queue.cc:1035] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:36.666751 31026 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: TxnStatusTablet state changed. Reason: New leader 03e96693820643c2bf8289f1ab417f20. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.666869 31026 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.667548 31038 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: TxnStatusTablet state changed. Reason: New leader 03e96693820643c2bf8289f1ab417f20. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.667675 31038 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.668223 31025 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.668359 31025 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:36.668962 31060 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.669076 31060 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 03e96693820643c2bf8289f1ab417f20: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:53:36.669796 31032 mvcc.cc:205] Tried to move back new op lower bound from 7298465860251586560 to 7298465859929415680. Current Snapshot: MvccSnapshot[applied={T|T < 7298465860251586560}]
I20260619 06:53:36.669942 31038 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.670022 31038 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P e13c15c9c8ee4c99843b284686efe607: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.672705 31033 mvcc.cc:205] Tried to move back new op lower bound from 7298465860251586560 to 7298465859929415680. Current Snapshot: MvccSnapshot[applied={T|T < 7298465860251586560 or (T in {7298465860251586560})}]
I20260619 06:53:36.672945 31026 tablet_replica.cc:439] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e96693820643c2bf8289f1ab417f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } } }
I20260619 06:53:36.673055 31026 tablet_replica.cc:442] T ef217df3dafa46c594cb5406db3ed053 P 101f8f50a6f54a6fa3d52b3845e929ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:53:36.685297 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.681882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.22.113.3:53859)
0619 06:53:36.682173 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.682186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.682318 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.682586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.682588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.682596 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:36.682712 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.682717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.683515 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.683518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.684135 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.684143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.684596 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.684599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.684685 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.684724 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.685023 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.685032 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.685093 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.685168 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260619 06:53:36.685293 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.681974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:53859 (local address 127.22.113.1:45675)
0619 06:53:36.682127 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.682132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.682258 (+ 126us) server_negotiation.cc:407] Connection header received
0619 06:53:36.682419 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.682423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.682476 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.682569 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:36.682813 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.683404 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.684252 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.684485 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.684593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.684858 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.684932 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.685022 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.685112 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.685158 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:53:36.713188 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.709024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38077 (local address 127.22.113.1:40777)
0619 06:53:36.709279 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:36.709298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:36.709404 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:36.709712 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:36.709714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:36.709719 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:53:36.709866 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.709871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.710945 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.710948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.711729 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:36.711738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.712399 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.712402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:36.712506 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.712562 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.712909 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.712915 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:53:36.712976 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:53:36.713051 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260619 06:53:36.713238 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.709368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40777 (local address 127.22.113.3:38077)
0619 06:53:36.709492 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.709497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.709513 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:36.709563 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.709566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.709619 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.709710 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:36.709982 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.710806 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.711858 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.712214 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.712313 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.712703 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:36.712809 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:36.712890 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.712991 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.713057 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:53:36.737995 30508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:36.738173 30508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:36.740274 30508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Sending DeleteTablet for 3 replicas of tablet 3a7e51af8d914ee9ab3452452eb2fa40
I20260619 06:53:36.740932 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:58068
I20260619 06:53:36.740964 30785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:36382
I20260619 06:53:36.741331 31077 tablet_replica.cc:330] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: stopping tablet replica
I20260619 06:53:36.741495 31077 raft_consensus.cc:2229] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:36.741587 31078 tablet_replica.cc:330] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: stopping tablet replica
I20260619 06:53:36.741585 30925 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7e51af8d914ee9ab3452452eb2fa40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:36 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260619 06:53:36.741619 31077 raft_consensus.cc:2258] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:36.741899 31079 tablet_replica.cc:330] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: stopping tablet replica
I20260619 06:53:36.741969 31078 raft_consensus.cc:2229] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:36.742013 31079 raft_consensus.cc:2229] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:36.742121 31078 raft_consensus.cc:2258] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:36.742204 31079 raft_consensus.cc:2258] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:36.742918 31079 ts_tablet_manager.cc:1905] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.743034 31077 ts_tablet_manager.cc:1905] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.744534 31078 ts_tablet_manager.cc:1905] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:36.746665 31079 ts_tablet_manager.cc:1918] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:36.746752 31079 log.cc:1196] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/3a7e51af8d914ee9ab3452452eb2fa40
I20260619 06:53:36.747033 31079 ts_tablet_manager.cc:1939] T 3a7e51af8d914ee9ab3452452eb2fa40 P e13c15c9c8ee4c99843b284686efe607: Deleting consensus metadata
I20260619 06:53:36.748989 30494 catalog_manager.cc:4841] TS e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077): tablet 3a7e51af8d914ee9ab3452452eb2fa40 (table test-table [id=fdd24999ffa54c15b63af127c0396fcf]) successfully deleted
I20260619 06:53:36.750065 31078 ts_tablet_manager.cc:1918] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:36.750156 31078 log.cc:1196] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/3a7e51af8d914ee9ab3452452eb2fa40
I20260619 06:53:36.750437 31078 ts_tablet_manager.cc:1939] T 3a7e51af8d914ee9ab3452452eb2fa40 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting consensus metadata
I20260619 06:53:36.751356 30492 catalog_manager.cc:4841] TS 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215): tablet 3a7e51af8d914ee9ab3452452eb2fa40 (table test-table [id=fdd24999ffa54c15b63af127c0396fcf]) successfully deleted
I20260619 06:53:36.751885 31077 ts_tablet_manager.cc:1918] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:36.751992 31077 log.cc:1196] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/3a7e51af8d914ee9ab3452452eb2fa40
I20260619 06:53:36.752265 31077 ts_tablet_manager.cc:1939] T 3a7e51af8d914ee9ab3452452eb2fa40 P 03e96693820643c2bf8289f1ab417f20: Deleting consensus metadata
I20260619 06:53:36.753047 30493 catalog_manager.cc:4841] TS 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675): tablet 3a7e51af8d914ee9ab3452452eb2fa40 (table test-table [id=fdd24999ffa54c15b63af127c0396fcf]) successfully deleted
Jun 19 06:53:36 dist-test-slave-g8zg krb5kdc[30463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:53:36 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:36 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:53:36.775764 31087 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:36.768523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.22.113.62:41421)
0619 06:53:36.768818 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.768823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.768845 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:53:36.768913 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.768917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.768982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.769132 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:36.769838 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.770670 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.771730 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.771980 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.772080 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.775570 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60756: BlockingRecv error: recv got EOF from 127.0.0.1:60756 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
Jun 19 06:53:36 dist-test-slave-g8zg krb5kdc[30463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM
I20260619 06:53:36.786226 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:36.777335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.22.113.62:41421)
0619 06:53:36.777511 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:36.777517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:36.777642 (+ 125us) server_negotiation.cc:407] Connection header received
0619 06:53:36.777799 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:36.777803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:36.777862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:36.777951 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:36.778563 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.779109 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.779940 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:36.780203 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:36.780325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:36.782953 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:36.782956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:36.782959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:36.783001 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:36.784469 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:36.784935 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:36.784939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:36.784941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:36.785056 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:36.785348 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:36.785351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:36.785352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:36.785549 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:36.785674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:36.785974 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 06:53:36.786081 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:53:36.789119 30508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:53:36.789598 30508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:36.794929 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 073d739a50084c708d215dd47e8edf98 (DEFAULT_TABLE table=test-table [id=c46a180276ff4f5eb5dfb632221561fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.795040 30925 tablet_service.cc:1467] Processing CreateTablet for tablet 073d739a50084c708d215dd47e8edf98 (DEFAULT_TABLE table=test-table [id=c46a180276ff4f5eb5dfb632221561fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.795027 30785 tablet_service.cc:1467] Processing CreateTablet for tablet 073d739a50084c708d215dd47e8edf98 (DEFAULT_TABLE table=test-table [id=c46a180276ff4f5eb5dfb632221561fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:36.795316 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073d739a50084c708d215dd47e8edf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.795316 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073d739a50084c708d215dd47e8edf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.795467 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073d739a50084c708d215dd47e8edf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:36.797955 31094 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Bootstrap starting.
I20260619 06:53:36.798349 31095 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Bootstrap starting.
I20260619 06:53:36.798954 31094 tablet_bootstrap.cc:654] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.799371 31095 tablet_bootstrap.cc:654] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.799803 31093 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Bootstrap starting.
I20260619 06:53:36.800701 31093 tablet_bootstrap.cc:654] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:36.801690 31094 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: No bootstrap required, opened a new log
I20260619 06:53:36.801801 31094 ts_tablet_manager.cc:1397] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 06:53:36.802093 31093 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: No bootstrap required, opened a new log
I20260619 06:53:36.802193 31093 ts_tablet_manager.cc:1397] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:53:36.802361 31094 raft_consensus.cc:348] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.802506 31094 raft_consensus.cc:374] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.802541 31094 raft_consensus.cc:729] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.802655 31093 raft_consensus.cc:348] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.802697 31094 consensus_queue.cc:260] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.802774 31093 raft_consensus.cc:374] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.802821 31093 raft_consensus.cc:729] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.803052 31094 ts_tablet_manager.cc:1428] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:36.803016 31093 consensus_queue.cc:260] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.803369 31093 ts_tablet_manager.cc:1428] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:53:36.803998 31095 tablet_bootstrap.cc:492] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: No bootstrap required, opened a new log
I20260619 06:53:36.804122 31095 ts_tablet_manager.cc:1397] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260619 06:53:36.804920 31095 raft_consensus.cc:348] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.805051 31095 raft_consensus.cc:374] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:36.805094 31095 raft_consensus.cc:729] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:36.805274 31095 consensus_queue.cc:260] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:36.805567 31095 ts_tablet_manager.cc:1428] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:37.106540 31026 raft_consensus.cc:482] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:37.106693 31026 raft_consensus.cc:504] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.107854 31026 leader_election.cc:288] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077)
I20260619 06:53:37.112640 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.108256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38077 (local address 127.22.113.2:52061)
0619 06:53:37.108642 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:37.108661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:37.108795 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:37.109079 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:37.109082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:37.109101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:37.109242 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:37.109250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.110129 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.110132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:37.111187 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:37.111196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.112186 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.112189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:37.112263 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.112289 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:53:37.112389 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:53:37.112472 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":96,"thread_start_us":133,"threads_started":1}
I20260619 06:53:37.112715 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.108351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:52061 (local address 127.22.113.3:38077)
0619 06:53:37.108491 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:37.108496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:37.108788 (+ 292us) server_negotiation.cc:407] Connection header received
0619 06:53:37.108897 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:37.108901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:37.108968 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:37.109060 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:37.109357 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.109989 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.111417 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.112055 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.112194 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.112268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:37.112422 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:53:37.112587 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:53:37.112844 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.108396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57651 (local address 127.22.113.1:45675)
0619 06:53:37.108566 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:37.108571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:37.108920 (+ 349us) server_negotiation.cc:407] Connection header received
0619 06:53:37.109076 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:37.109080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:37.109132 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:37.109221 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:53:37.109610 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.110189 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.111541 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.112187 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.112274 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.112351 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:37.112614 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 06:53:37.112697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:53:37.112879 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.108256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.22.113.2:57651)
0619 06:53:37.108779 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:37.108794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:37.108933 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:37.109249 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:37.109251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:37.109266 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:53:37.109438 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:37.109445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.110350 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.110353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:37.111366 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:37.111380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.112388 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.112392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:37.112485 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.112503 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:53:37.112596 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:53:37.112685 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":76,"thread_start_us":103,"threads_started":1}
I20260619 06:53:37.113500 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073d739a50084c708d215dd47e8edf98" candidate_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13c15c9c8ee4c99843b284686efe607" is_pre_election: true
I20260619 06:53:37.113593 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073d739a50084c708d215dd47e8edf98" candidate_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20" is_pre_election: true
I20260619 06:53:37.113703 30945 raft_consensus.cc:2454] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101f8f50a6f54a6fa3d52b3845e929ec in term 0.
I20260619 06:53:37.113770 30665 raft_consensus.cc:2454] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101f8f50a6f54a6fa3d52b3845e929ec in term 0.
I20260619 06:53:37.114207 30738 leader_election.cc:302] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:37.114609 31026 raft_consensus.cc:2781] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:37.114670 31026 raft_consensus.cc:482] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:37.114699 31026 raft_consensus.cc:3037] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.115952 31026 raft_consensus.cc:504] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.116348 31026 leader_election.cc:288] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [CANDIDATE]: Term 1 election: Requested vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077)
I20260619 06:53:37.116712 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073d739a50084c708d215dd47e8edf98" candidate_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20"
I20260619 06:53:37.116847 30665 raft_consensus.cc:3037] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.116796 30945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073d739a50084c708d215dd47e8edf98" candidate_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13c15c9c8ee4c99843b284686efe607"
I20260619 06:53:37.116967 30945 raft_consensus.cc:3037] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.117872 30665 raft_consensus.cc:2454] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101f8f50a6f54a6fa3d52b3845e929ec in term 1.
I20260619 06:53:37.118163 30737 leader_election.cc:302] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e96693820643c2bf8289f1ab417f20, 101f8f50a6f54a6fa3d52b3845e929ec; no voters:
I20260619 06:53:37.118196 30945 raft_consensus.cc:2454] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101f8f50a6f54a6fa3d52b3845e929ec in term 1.
I20260619 06:53:37.118340 31026 raft_consensus.cc:2781] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:37.118541 31026 raft_consensus.cc:686] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 LEADER]: Becoming Leader. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Running, Role: LEADER
I20260619 06:53:37.118822 31026 consensus_queue.cc:237] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.121555 30506 catalog_manager.cc:5495] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2). New cstate: current_term: 1 leader_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } health_report { overall_health: UNKNOWN } } }
I20260619 06:53:37.168676 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.164835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.22.113.2:41215)
0619 06:53:37.165188 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:37.165195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:37.165208 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:53:37.165284 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:37.165286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:37.165341 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:37.165471 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:37.165715 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.166249 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.167105 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.167407 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.167526 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.167805 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:37.168249 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:37.168369 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:37.168438 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:53:37.168489 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
I20260619 06:53:37.171788 30665 raft_consensus.cc:1261] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Refusing update from remote peer 101f8f50a6f54a6fa3d52b3845e929ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:37.171808 30945 raft_consensus.cc:1261] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Refusing update from remote peer 101f8f50a6f54a6fa3d52b3845e929ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:37.172420 31026 consensus_queue.cc:1035] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:37.172678 31101 consensus_queue.cc:1035] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:37.185515 30506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:37.185694 30506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:37.187947 30506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Sending DeleteTablet for 3 replicas of tablet 073d739a50084c708d215dd47e8edf98
I20260619 06:53:37.188591 30925 tablet_service.cc:1514] Processing DeleteTablet for tablet 073d739a50084c708d215dd47e8edf98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260619 06:53:37.188602 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 073d739a50084c708d215dd47e8edf98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:58068
I20260619 06:53:37.188772 31079 tablet_replica.cc:330] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: stopping tablet replica
I20260619 06:53:37.188709 30785 tablet_service.cc:1514] Processing DeleteTablet for tablet 073d739a50084c708d215dd47e8edf98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:36382
I20260619 06:53:37.188884 31079 raft_consensus.cc:2229] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:37.188921 31078 tablet_replica.cc:330] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: stopping tablet replica
I20260619 06:53:37.189044 31079 raft_consensus.cc:2258] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.189083 31078 raft_consensus.cc:2229] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:37.189224 31077 tablet_replica.cc:330] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: stopping tablet replica
I20260619 06:53:37.189371 31077 raft_consensus.cc:2229] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:37.189436 31078 raft_consensus.cc:2258] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.189528 31077 raft_consensus.cc:2258] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.190346 31077 ts_tablet_manager.cc:1905] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.190459 31079 ts_tablet_manager.cc:1905] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.191118 31078 ts_tablet_manager.cc:1905] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.193001 31077 ts_tablet_manager.cc:1918] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:37.193064 31077 log.cc:1196] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/073d739a50084c708d215dd47e8edf98
I20260619 06:53:37.193354 31077 ts_tablet_manager.cc:1939] T 073d739a50084c708d215dd47e8edf98 P 03e96693820643c2bf8289f1ab417f20: Deleting consensus metadata
I20260619 06:53:37.193495 31079 ts_tablet_manager.cc:1918] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:37.193562 31079 log.cc:1196] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/073d739a50084c708d215dd47e8edf98
I20260619 06:53:37.193852 31079 ts_tablet_manager.cc:1939] T 073d739a50084c708d215dd47e8edf98 P e13c15c9c8ee4c99843b284686efe607: Deleting consensus metadata
I20260619 06:53:37.193902 31078 ts_tablet_manager.cc:1918] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:53:37.193984 31078 log.cc:1196] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/073d739a50084c708d215dd47e8edf98
I20260619 06:53:37.194247 31078 ts_tablet_manager.cc:1939] T 073d739a50084c708d215dd47e8edf98 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting consensus metadata
I20260619 06:53:37.194245 30493 catalog_manager.cc:4841] TS 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675): tablet 073d739a50084c708d215dd47e8edf98 (table test-table [id=c46a180276ff4f5eb5dfb632221561fe]) successfully deleted
I20260619 06:53:37.194566 30506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:53:37.194615 30494 catalog_manager.cc:4841] TS e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077): tablet 073d739a50084c708d215dd47e8edf98 (table test-table [id=c46a180276ff4f5eb5dfb632221561fe]) successfully deleted
I20260619 06:53:37.194972 30492 catalog_manager.cc:4841] TS 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215): tablet 073d739a50084c708d215dd47e8edf98 (table test-table [id=c46a180276ff4f5eb5dfb632221561fe]) successfully deleted
W20260619 06:53:37.195026 30506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:53:37.199753 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 535285862e3b415fb242431a6032d325 (DEFAULT_TABLE table=test-table [id=8fac6ba9a2804862a7e31bca1184ea42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:37.200029 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535285862e3b415fb242431a6032d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:37.200064 30925 tablet_service.cc:1467] Processing CreateTablet for tablet 535285862e3b415fb242431a6032d325 (DEFAULT_TABLE table=test-table [id=8fac6ba9a2804862a7e31bca1184ea42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:37.200330 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535285862e3b415fb242431a6032d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:37.200421 30785 tablet_service.cc:1467] Processing CreateTablet for tablet 535285862e3b415fb242431a6032d325 (DEFAULT_TABLE table=test-table [id=8fac6ba9a2804862a7e31bca1184ea42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:53:37.200703 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535285862e3b415fb242431a6032d325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:37.202450 31093 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: Bootstrap starting.
I20260619 06:53:37.202890 31094 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Bootstrap starting.
I20260619 06:53:37.203354 31093 tablet_bootstrap.cc:654] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:37.203482 31095 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Bootstrap starting.
I20260619 06:53:37.203809 31094 tablet_bootstrap.cc:654] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:37.204567 31095 tablet_bootstrap.cc:654] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:37.204625 31093 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: No bootstrap required, opened a new log
I20260619 06:53:37.204701 31093 ts_tablet_manager.cc:1397] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:53:37.205075 31093 raft_consensus.cc:348] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.205214 31093 raft_consensus.cc:374] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:37.205255 31093 raft_consensus.cc:729] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e96693820643c2bf8289f1ab417f20, State: Initialized, Role: FOLLOWER
I20260619 06:53:37.205385 31093 consensus_queue.cc:260] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.205706 31093 ts_tablet_manager.cc:1428] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:37.206100 31095 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: No bootstrap required, opened a new log
I20260619 06:53:37.206188 31095 ts_tablet_manager.cc:1397] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:53:37.206712 31095 raft_consensus.cc:348] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.206844 31095 raft_consensus.cc:374] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:37.206877 31095 raft_consensus.cc:729] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8f50a6f54a6fa3d52b3845e929ec, State: Initialized, Role: FOLLOWER
I20260619 06:53:37.207050 31095 consensus_queue.cc:260] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.207446 31095 ts_tablet_manager.cc:1428] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:53:37.208774 31094 tablet_bootstrap.cc:492] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: No bootstrap required, opened a new log
I20260619 06:53:37.208894 31094 ts_tablet_manager.cc:1397] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 06:53:37.209318 31094 raft_consensus.cc:348] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.209426 31094 raft_consensus.cc:374] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:37.209465 31094 raft_consensus.cc:729] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Initialized, Role: FOLLOWER
I20260619 06:53:37.209627 31094 consensus_queue.cc:260] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.209904 31094 ts_tablet_manager.cc:1428] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:53:37.299366 31038 raft_consensus.cc:482] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:53:37.299584 31038 raft_consensus.cc:504] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.300038 31038 leader_election.cc:288] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215)
I20260619 06:53:37.300661 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535285862e3b415fb242431a6032d325" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" is_pre_election: true
I20260619 06:53:37.300766 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535285862e3b415fb242431a6032d325" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20" is_pre_election: true
I20260619 06:53:37.300832 30804 raft_consensus.cc:2454] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:37.300899 30665 raft_consensus.cc:2454] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 0.
I20260619 06:53:37.301222 30876 leader_election.cc:302] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:37.301501 31038 raft_consensus.cc:2781] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:53:37.301586 31038 raft_consensus.cc:482] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:53:37.301625 31038 raft_consensus.cc:3037] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.302716 31038 raft_consensus.cc:504] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.303077 31038 leader_election.cc:288] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Requested vote from peers 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675), 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215)
I20260619 06:53:37.303503 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535285862e3b415fb242431a6032d325" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8f50a6f54a6fa3d52b3845e929ec"
I20260619 06:53:37.303501 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535285862e3b415fb242431a6032d325" candidate_uuid: "e13c15c9c8ee4c99843b284686efe607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e96693820643c2bf8289f1ab417f20"
I20260619 06:53:37.303633 30804 raft_consensus.cc:3037] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.303700 30665 raft_consensus.cc:3037] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:37.304657 30804 raft_consensus.cc:2454] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:37.304939 30876 leader_election.cc:302] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101f8f50a6f54a6fa3d52b3845e929ec, e13c15c9c8ee4c99843b284686efe607; no voters:
I20260619 06:53:37.305150 31038 raft_consensus.cc:2781] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:37.305320 31038 raft_consensus.cc:686] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Becoming Leader. State: Replica: e13c15c9c8ee4c99843b284686efe607, State: Running, Role: LEADER
I20260619 06:53:37.305388 30665 raft_consensus.cc:2454] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c15c9c8ee4c99843b284686efe607 in term 1.
I20260619 06:53:37.305471 31038 consensus_queue.cc:237] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } }
I20260619 06:53:37.307550 30506 catalog_manager.cc:5495] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13c15c9c8ee4c99843b284686efe607 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "e13c15c9c8ee4c99843b284686efe607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13c15c9c8ee4c99843b284686efe607" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38077 } health_report { overall_health: HEALTHY } } }
I20260619 06:53:37.345214 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.341608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.22.113.3:38077)
0619 06:53:37.341739 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:37.341743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:37.341758 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:53:37.341859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:37.341864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:37.341979 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:37.342101 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:53:37.342368 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.342923 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.343872 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:37.344096 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:37.344233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:37.344540 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:53:37.344674 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:53:37.344797 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:37.344951 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:53:37.345033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:53:37.352591 30665 raft_consensus.cc:1261] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Refusing update from remote peer e13c15c9c8ee4c99843b284686efe607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:37.352762 30804 raft_consensus.cc:1261] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Refusing update from remote peer e13c15c9c8ee4c99843b284686efe607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:53:37.353165 31038 consensus_queue.cc:1035] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101f8f50a6f54a6fa3d52b3845e929ec" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:37.353324 31111 consensus_queue.cc:1035] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e96693820643c2bf8289f1ab417f20" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:53:37.388722 30506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:37.388983 30506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:53:37.391203 30506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19f50a85faf44438b02769f4abedd21: Sending DeleteTablet for 3 replicas of tablet 535285862e3b415fb242431a6032d325
I20260619 06:53:37.391995 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 535285862e3b415fb242431a6032d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:58068
I20260619 06:53:37.392061 30925 tablet_service.cc:1514] Processing DeleteTablet for tablet 535285862e3b415fb242431a6032d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59868
I20260619 06:53:37.392158 30785 tablet_service.cc:1514] Processing DeleteTablet for tablet 535285862e3b415fb242431a6032d325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:53:37 UTC) from {username='oryx', principal='oryx/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:36382
I20260619 06:53:37.392225 31077 tablet_replica.cc:330] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: stopping tablet replica
I20260619 06:53:37.392252 31079 tablet_replica.cc:330] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: stopping tablet replica
I20260619 06:53:37.392357 31078 tablet_replica.cc:330] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: stopping tablet replica
I20260619 06:53:37.392369 31077 raft_consensus.cc:2229] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:37.392379 31079 raft_consensus.cc:2229] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:53:37.392493 31077 raft_consensus.cc:2258] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.392496 31078 raft_consensus.cc:2229] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:53:37.392585 31079 raft_consensus.cc:2258] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.392633 31078 raft_consensus.cc:2258] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:53:37.393431 31079 ts_tablet_manager.cc:1905] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.393467 31077 ts_tablet_manager.cc:1905] T 535285862e3b415fb242431a6032d325 P 03e96693820643c2bf8289f1ab417f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.393471 31078 ts_tablet_manager.cc:1905] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:53:37.394088 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30580
I20260619 06:53:37.397853 31078 ts_tablet_manager.cc:1918] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:37.397940 31078 log.cc:1196] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/535285862e3b415fb242431a6032d325
I20260619 06:53:37.398260 31078 ts_tablet_manager.cc:1939] T 535285862e3b415fb242431a6032d325 P 101f8f50a6f54a6fa3d52b3845e929ec: Deleting consensus metadata
I20260619 06:53:37.399030 31079 ts_tablet_manager.cc:1918] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:53:37.399092 30492 catalog_manager.cc:4841] TS 101f8f50a6f54a6fa3d52b3845e929ec (127.22.113.2:41215): tablet 535285862e3b415fb242431a6032d325 (table test-table [id=8fac6ba9a2804862a7e31bca1184ea42]) successfully deleted
I20260619 06:53:37.399155 31079 log.cc:1196] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/535285862e3b415fb242431a6032d325
I20260619 06:53:37.399529 31079 ts_tablet_manager.cc:1939] T 535285862e3b415fb242431a6032d325 P e13c15c9c8ee4c99843b284686efe607: Deleting consensus metadata
I20260619 06:53:37.400244 30494 catalog_manager.cc:4841] TS e13c15c9c8ee4c99843b284686efe607 (127.22.113.3:38077): tablet 535285862e3b415fb242431a6032d325 (table test-table [id=8fac6ba9a2804862a7e31bca1184ea42]) successfully deleted
W20260619 06:53:37.406211 30493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:45675 (error 108)
I20260619 06:53:37.406354 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30720
I20260619 06:53:37.408648 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:37.407888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:45675 (local address 127.0.0.1:58096)
0619 06:53:37.408369 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:37.408481 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:45675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":65,"threads_started":1}
W20260619 06:53:37.408856 30493 catalog_manager.cc:4584] TS 03e96693820643c2bf8289f1ab417f20 (127.22.113.1:45675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 535285862e3b415fb242431a6032d325: Network error: Client connection negotiation failed: client connection to 127.22.113.1:45675: connect: Connection refused (error 111)
I20260619 06:53:37.414839 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30860
I20260619 06:53:37.422649 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 30476
2026-06-19T06:53:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3693 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:37 dist-test-slave-g8zg krb5kdc[31125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:37 dist-test-slave-g8zg krb5kdc[31125](info): set up 2 sockets
Jun 19 06:53:37 dist-test-slave-g8zg krb5kdc[31125](info): commencing operation
krb5kdc: starting...
W20260619 06:53:38.516127 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:38 dist-test-slave-g8zg krb5kdc[31125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:38.651356 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36567
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:40839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:36567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:38.765214 31138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:38.765533 31138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:38.765609 31138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:38.769619 31138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:38.769716 31138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:38.769754 31138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:38.769793 31138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:38.769829 31138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:38.775106 31138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:40839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:36567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:38.776204 31138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:38.777256 31138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:38.786442 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:38.786686 31138 server_base.cc:1034] running on GCE node
W20260619 06:53:38.786489 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:38.786478 31143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:38.787523 31138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:38.788689 31138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:38.789911 31138 hybrid_clock.cc:648] HybridClock initialized: now 1781852018789890 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:53:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1375 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:38 dist-test-slave-g8zg krb5kdc[31151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:38 dist-test-slave-g8zg krb5kdc[31151](info): set up 2 sockets
Jun 19 06:53:38 dist-test-slave-g8zg krb5kdc[31151](info): commencing operation
krb5kdc: starting...
W20260619 06:53:39.856166 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:53:39 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:53:39.988451 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32801
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:40.109678 31164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:40.109987 31164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:40.110029 31164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:40.114629 31164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:40.114710 31164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:40.114732 31164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:40.114750 31164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:40.114836 31164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:40.120592 31164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:40.121910 31164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:40.123144 31164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:40.133152 31169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.133206 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.133203 31164 server_base.cc:1034] running on GCE node
W20260619 06:53:40.133381 31170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.134182 31164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:40.135632 31164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:40.136829 31164 hybrid_clock.cc:648] HybridClock initialized: now 1781852020136794 us; error 51 us; skew 500 ppm
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:40.140007 31164 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:40.141041 31164 webserver.cc:460] Webserver started at http://127.22.113.62:41719/ using document root <none> and password file <none>
I20260619 06:53:40.141654 31164 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:40.141718 31164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:40.141918 31164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:40.144323 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "0f5da235de2c4632839596d25f6e0c8d"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "f16878f5664cf712e9700c4e4babb151"
server_key_iv: "44b6273036c2d5722e1957374bebc784"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.144970 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "0f5da235de2c4632839596d25f6e0c8d"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "f16878f5664cf712e9700c4e4babb151"
server_key_iv: "44b6273036c2d5722e1957374bebc784"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.150125 31164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:53:40.153139 31179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.154723 31164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:40.154897 31164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "0f5da235de2c4632839596d25f6e0c8d"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "f16878f5664cf712e9700c4e4babb151"
server_key_iv: "44b6273036c2d5722e1957374bebc784"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.155046 31164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:40.190258 31164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:40.193965 31164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:40.194202 31164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:40.203058 31164 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:32801
I20260619 06:53:40.203096 31231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:32801 every 8 connection(s)
I20260619 06:53:40.204211 31164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:40.206090 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31164
I20260619 06:53:40.206212 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:40.206594 22980 external_mini_cluster.cc:1428] Setting key db4252df4c66dd38c35a266461819b7b
I20260619 06:53:40.207438 31232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:40.214697 31232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Bootstrap starting.
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:40.217516 31232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:40.218441 31232 log.cc:824] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:40.221046 31232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: No bootstrap required, opened a new log
I20260619 06:53:40.223315 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:40.208262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.22.113.62:32801)
0619 06:53:40.208736 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:40.208760 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:40.208805 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:53:40.209579 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:40.209598 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:40.209973 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:40.210312 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:40.211376 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.212468 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.213736 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.214257 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.214425 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:40.217552 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:40.217560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:40.217578 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:40.217622 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:40.219700 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.220242 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.220247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.220253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.220384 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.220664 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.220667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.220669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.221221 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:40.221434 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:40.221752 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:53:40.222070 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":162,"threads_started":1}
I20260619 06:53:40.224753 31232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:40.224970 31232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:40.225031 31232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5da235de2c4632839596d25f6e0c8d, State: Initialized, Role: FOLLOWER
I20260619 06:53:40.225487 31232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:40.225615 31232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:40.225710 31232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:40.225837 31232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:40.226816 31232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:40.227181 31232 leader_election.cc:302] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f5da235de2c4632839596d25f6e0c8d; no voters:
I20260619 06:53:40.227566 31232 leader_election.cc:288] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:40.227770 31237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:40.228119 31237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 LEADER]: Becoming Leader. State: Replica: 0f5da235de2c4632839596d25f6e0c8d, State: Running, Role: LEADER
I20260619 06:53:40.228515 31237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:40.228880 31232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:40.230461 31239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f5da235de2c4632839596d25f6e0c8d. Latest consensus state: current_term: 1 leader_uuid: "0f5da235de2c4632839596d25f6e0c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } } }
I20260619 06:53:40.230604 31239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:40.230947 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f5da235de2c4632839596d25f6e0c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } } }
I20260619 06:53:40.231036 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: This master's current role is: LEADER
W20260619 06:53:40.234025 31252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:40.234112 31252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:40.234210 31250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:40.234972 31250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:40.241824 31250 catalog_manager.cc:1347] Generated new cluster ID: 3dd0300c329c441195205682a7c66f82
I20260619 06:53:40.241930 31250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:40.268249 31250 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:40.269270 31250 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:40.274871 31250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Generated new TSK 0
I20260619 06:53:40.276003 31250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:53:40.314611 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:40.432220 31258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:40.432523 31258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:40.432596 31258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:40.436642 31258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:40.436734 31258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:40.436831 31258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:40.442183 31258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:40.443311 31258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:40.444345 31258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:40.454525 31266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.454548 31263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.454582 31264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.454782 31258 server_base.cc:1034] running on GCE node
I20260619 06:53:40.455371 31258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:40.455989 31258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:40.457175 31258 hybrid_clock.cc:648] HybridClock initialized: now 1781852020457140 us; error 57 us; skew 500 ppm
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:40.460261 31258 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:53:40.461450 31258 webserver.cc:460] Webserver started at http://127.22.113.1:40531/ using document root <none> and password file <none>
I20260619 06:53:40.462152 31258 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:40.462218 31258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:40.462417 31258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:40.464818 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "79e3140c251c492ba048e7f6fcf1a5d5"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a8a024c40ebd8d163c0399e5bd379be8"
server_key_iv: "0fcf67597735b614265a7eec3ae9b766"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.465365 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "79e3140c251c492ba048e7f6fcf1a5d5"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a8a024c40ebd8d163c0399e5bd379be8"
server_key_iv: "0fcf67597735b614265a7eec3ae9b766"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.469609 31258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:40.472404 31273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.473991 31258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:40.474108 31258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "79e3140c251c492ba048e7f6fcf1a5d5"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a8a024c40ebd8d163c0399e5bd379be8"
server_key_iv: "0fcf67597735b614265a7eec3ae9b766"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.474203 31258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:40.514539 31258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:40.518055 31258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:40.518234 31258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:40.518867 31258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:40.519919 31258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:40.519994 31258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.520076 31258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:40.520118 31258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.531525 31258 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:35857
I20260619 06:53:40.531546 31386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:35857 every 8 connection(s)
I20260619 06:53:40.532562 31258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:40.533061 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31258
I20260619 06:53:40.533207 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:40.533612 22980 external_mini_cluster.cc:1428] Setting key 828a0eee2497a73c1629b3cf971db1c2
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:53:40.547806 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:40.534720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:39399 (local address 127.22.113.62:32801)
0619 06:53:40.534854 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:40.534857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:40.536099 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 06:53:40.537603 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:40.537607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:40.537661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:40.537752 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:40.539379 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.539932 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.541145 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.541369 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.541480 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:40.544023 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:40.544026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:40.544028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:40.544050 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:40.545631 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.546189 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.546197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.546199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.546269 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.546616 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.546624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.546625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.546828 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:40.546939 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:40.547491 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 06:53:40.547627 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:53:40.548588 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:40.535350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:39399)
0619 06:53:40.535941 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:40.535989 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:40.537327 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:40.537984 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:40.537996 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:40.538553 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:40.539180 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:40.539194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.540059 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.540070 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:40.541016 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:40.541025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.541542 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.541547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:40.541701 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:40.542312 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:40.542328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:40.543858 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:40.545748 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:40.545754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:40.545769 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:40.546074 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:40.546365 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:40.546368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:40.546370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:40.546506 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:40.546947 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:40.546956 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:40.547202 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 06:53:40.547437 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 06:53:40.547747 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":163,"threads_started":1}
I20260619 06:53:40.549903 31387 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32801
I20260619 06:53:40.550154 31387 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:40.550662 31387 heartbeater.cc:511] Master 127.22.113.62:32801 requested a full tablet report, sending...
I20260619 06:53:40.552651 31196 ts_manager.cc:194] Registered new tserver with Master: 79e3140c251c492ba048e7f6fcf1a5d5 (127.22.113.1:35857)
I20260619 06:53:40.554569 31196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:39399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:53:40.564004 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:40.675395 31392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:40.675710 31392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:40.675787 31392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:40.680059 31392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:40.680145 31392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:40.680238 31392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:40.685510 31392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:40.686553 31392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:40.687594 31392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:40.697417 31397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.697425 31400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.697448 31392 server_base.cc:1034] running on GCE node
W20260619 06:53:40.697484 31398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.698189 31392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:40.698832 31392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:40.700043 31392 hybrid_clock.cc:648] HybridClock initialized: now 1781852020699977 us; error 84 us; skew 500 ppm
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:40.702922 31392 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:53:40.703877 31392 webserver.cc:460] Webserver started at http://127.22.113.2:43723/ using document root <none> and password file <none>
I20260619 06:53:40.704456 31392 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:40.704546 31392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:40.704789 31392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:40.706818 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "21ac06c215444043a9eabbec12fe8242"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a54add6ef9059ce060ad985723019595"
server_key_iv: "baf7e3b02069d87c568f6ed8b07f1ba5"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.707435 31392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "21ac06c215444043a9eabbec12fe8242"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a54add6ef9059ce060ad985723019595"
server_key_iv: "baf7e3b02069d87c568f6ed8b07f1ba5"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.711457 31392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:40.714213 31407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.715848 31392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:40.716003 31392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "21ac06c215444043a9eabbec12fe8242"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a54add6ef9059ce060ad985723019595"
server_key_iv: "baf7e3b02069d87c568f6ed8b07f1ba5"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.716125 31392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:40.758561 31392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:40.761768 31392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:40.761936 31392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:40.762473 31392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:40.763396 31392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:40.763466 31392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.763531 31392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:40.763561 31392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.772861 31392 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:35379
I20260619 06:53:40.772977 31520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:35379 every 8 connection(s)
I20260619 06:53:40.773787 31392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:40.781580 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31392
I20260619 06:53:40.781744 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:40.782092 22980 external_mini_cluster.cc:1428] Setting key 8f60f744d32fb6ca4a87b27d092bbfbf
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:40.789300 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:40.776369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:54081 (local address 127.22.113.62:32801)
0619 06:53:40.776516 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:40.776520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:40.777654 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 06:53:40.779188 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:40.779192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:40.779271 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:40.779345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:40.780468 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.780985 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.782228 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.782459 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.782568 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:40.785553 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:40.785557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:40.785559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:40.785585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:40.787147 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.787726 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.787730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.787734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.787810 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:40.788153 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:40.788158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:40.788163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:40.788425 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:40.788538 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:40.789007 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 06:53:40.789129 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:53:40.790151 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:40.776846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:54081)
0619 06:53:40.777397 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:40.777470 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:40.778972 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:40.779516 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:40.779524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:40.779884 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:40.780332 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:40.780344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.781104 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.781108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:40.782099 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:40.782108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:40.782617 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:40.782621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:40.782835 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:40.783505 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:40.783521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:40.785396 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:40.787315 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:40.787322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:40.787337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:40.787605 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:40.787904 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:40.787908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:40.787910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:40.788056 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:40.788537 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:40.788544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:40.788781 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 06:53:40.788967 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 06:53:40.789278 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":166,"threads_started":1}
I20260619 06:53:40.791816 31521 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32801
I20260619 06:53:40.792073 31521 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:53:40.792564 31521 heartbeater.cc:511] Master 127.22.113.62:32801 requested a full tablet report, sending...
I20260619 06:53:40.793748 31196 ts_manager.cc:194] Registered new tserver with Master: 21ac06c215444043a9eabbec12fe8242 (127.22.113.2:35379)
I20260619 06:53:40.794404 31196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:54081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:53:40.812013 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:40.932869 31526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:40.933161 31526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:40.933207 31526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:40.937175 31526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:40.937239 31526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:40.937297 31526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:40.942625 31526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:40.943830 31526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:40.945007 31526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:40.956209 31531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.956203 31534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:40.956202 31532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:40.956362 31526 server_base.cc:1034] running on GCE node
I20260619 06:53:40.956984 31526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:40.957710 31526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:40.958907 31526 hybrid_clock.cc:648] HybridClock initialized: now 1781852020958866 us; error 60 us; skew 500 ppm
Jun 19 06:53:40 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:40.962697 31526 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:53:40.963816 31526 webserver.cc:460] Webserver started at http://127.22.113.3:39773/ using document root <none> and password file <none>
I20260619 06:53:40.964500 31526 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:40.964596 31526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:40.964843 31526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:40.967171 31526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "2584e98fb19e43059a91bb369cfdc89e"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a921fbfac348a228b681c03eca1239a1"
server_key_iv: "f1db29c2e33d0f2fa19b6dd17161003d"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.967849 31526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "2584e98fb19e43059a91bb369cfdc89e"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a921fbfac348a228b681c03eca1239a1"
server_key_iv: "f1db29c2e33d0f2fa19b6dd17161003d"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.972203 31526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:40.975160 31541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:40.976770 31526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:53:40.976919 31526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "2584e98fb19e43059a91bb369cfdc89e"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a921fbfac348a228b681c03eca1239a1"
server_key_iv: "f1db29c2e33d0f2fa19b6dd17161003d"
server_key_version: "encryptionkey@0"
I20260619 06:53:40.977038 31526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:41.000083 31526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:41.003654 31526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:41.003899 31526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:41.004621 31526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:41.005617 31526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:41.005692 31526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.005769 31526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:41.005817 31526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.017534 31526 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:36589
I20260619 06:53:41.017572 31654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:36589 every 8 connection(s)
I20260619 06:53:41.018707 31526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:41.019596 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31526
I20260619 06:53:41.019767 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:41.020098 22980 external_mini_cluster.cc:1428] Setting key 830bd1d0e96288029cabea14e038138b
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852020, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:53:41.034192 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.021134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:41331 (local address 127.22.113.62:32801)
0619 06:53:41.021289 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.021293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.022276 (+ 983us) server_negotiation.cc:407] Connection header received
0619 06:53:41.023745 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.023748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.023804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.023913 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.025173 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.025737 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.026967 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.027198 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.027436 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.030107 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:41.030110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:41.030113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:41.030138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:41.031826 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:41.032392 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:41.032396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:41.032398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:41.032485 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:41.032824 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:41.032828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:41.032831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:41.033091 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:41.033198 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:41.033851 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 06:53:41.034000 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:53:41.035024 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.021585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:41331)
0619 06:53:41.022089 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.022145 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.023507 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.024081 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.024088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.024451 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.025027 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.025039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.025877 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.025880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.026828 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.026837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.027481 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.027484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.027632 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.028266 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.028282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.029944 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:41.031962 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:41.031967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:41.031978 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:41.032286 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:41.032580 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:53:41.032582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:53:41.032583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:53:41.032714 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:53:41.033220 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:41.033227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:41.033552 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 06:53:41.033806 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 06:53:41.034105 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":168,"threads_started":1}
I20260619 06:53:41.036509 31655 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32801
I20260619 06:53:41.036780 31655 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:41.037288 31655 heartbeater.cc:511] Master 127.22.113.62:32801 requested a full tablet report, sending...
I20260619 06:53:41.038465 31196 ts_manager.cc:194] Registered new tserver with Master: 2584e98fb19e43059a91bb369cfdc89e (127.22.113.3:36589)
I20260619 06:53:41.039098 31196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:41331
I20260619 06:53:41.046590 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
Entry for principal oryx/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab.
WARNING: no policy specified for oryx/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
Entry for principal oryx/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab.
WARNING: no policy specified for oryx/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
Entry for principal oryx/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab.
I20260619 06:53:41.127094 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31258
I20260619 06:53:41.133471 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31392
I20260619 06:53:41.140138 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31526
I20260619 06:53:41.146564 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31164
I20260619 06:53:41.153985 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32801
--webserver_interface=127.22.113.62
--webserver_port=41719
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:53:41.275822 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:41.276132 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:41.276175 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:41.280588 31664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:41.280682 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:41.280721 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:41.280756 31664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:41.280784 31664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:41.286825 31664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:41.288087 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:41.289225 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:41.299093 31669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.299136 31664 server_base.cc:1034] running on GCE node
W20260619 06:53:41.299129 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.299221 31670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.299954 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:41.301199 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:41.302372 31664 hybrid_clock.cc:648] HybridClock initialized: now 1781852021302340 us; error 49 us; skew 500 ppm
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852021, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:41.305249 31664 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:53:41.306254 31664 webserver.cc:460] Webserver started at http://127.22.113.62:41719/ using document root <none> and password file <none>
I20260619 06:53:41.306901 31664 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:41.307018 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:41.311431 31664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:41.313889 31679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.315346 31664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:41.315485 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "0f5da235de2c4632839596d25f6e0c8d"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "f16878f5664cf712e9700c4e4babb151"
server_key_iv: "44b6273036c2d5722e1957374bebc784"
server_key_version: "encryptionkey@0"
I20260619 06:53:41.316124 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:41.341687 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:41.345444 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:41.345700 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:41.354645 31664 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:32801
I20260619 06:53:41.354676 31731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:32801 every 8 connection(s)
I20260619 06:53:41.355899 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:41.360509 31732 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:53:41.362129 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31664
I20260619 06:53:41.362437 31732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Bootstrap starting.
I20260619 06:53:41.362638 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:35857
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:32801
--webserver_port=40531
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:53:41.379678 31732 log.cc:824] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:41.386140 31732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:53:41.386510 31732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Bootstrap complete.
I20260619 06:53:41.390908 31732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:41.391397 31732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5da235de2c4632839596d25f6e0c8d, State: Initialized, Role: FOLLOWER
I20260619 06:53:41.391922 31732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:41.392061 31732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:41.392153 31732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:41.392259 31732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:53:41.394251 31732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:41.394589 31732 leader_election.cc:302] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f5da235de2c4632839596d25f6e0c8d; no voters:
I20260619 06:53:41.394914 31732 leader_election.cc:288] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:53:41.395205 31736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:53:41.395579 31736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [term 2 LEADER]: Becoming Leader. State: Replica: 0f5da235de2c4632839596d25f6e0c8d, State: Running, Role: LEADER
I20260619 06:53:41.395982 31736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } }
I20260619 06:53:41.396202 31732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:41.397554 31738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f5da235de2c4632839596d25f6e0c8d. Latest consensus state: current_term: 2 leader_uuid: "0f5da235de2c4632839596d25f6e0c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } } }
I20260619 06:53:41.397733 31738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:41.398097 31737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f5da235de2c4632839596d25f6e0c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f5da235de2c4632839596d25f6e0c8d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32801 } } }
I20260619 06:53:41.398200 31737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:41.398162 31749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:41.401482 31749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:41.402032 31749 catalog_manager.cc:1259] Loaded cluster ID: 3dd0300c329c441195205682a7c66f82
I20260619 06:53:41.402103 31749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:41.403878 31749 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:41.404428 31749 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f5da235de2c4632839596d25f6e0c8d: Loaded TSK: 0
I20260619 06:53:41.405412 31749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:53:41.499352 31734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:41.499614 31734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:41.499665 31734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:41.503690 31734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:41.503748 31734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:41.503806 31734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:41.509166 31734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.1.keytab
--principal=oryx/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:35857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=40531
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:41.510246 31734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:41.511344 31734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:41.521040 31761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.521107 31759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.521175 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.522293 31734 server_base.cc:1034] running on GCE node
I20260619 06:53:41.522787 31734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:41.523488 31734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:41.524721 31734 hybrid_clock.cc:648] HybridClock initialized: now 1781852021524694 us; error 58 us; skew 500 ppm
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852021, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:41.528028 31734 init.cc:377] Logged in from keytab as oryx/127.22.113.1@KRBTEST.COM (short username oryx)
I20260619 06:53:41.529011 31734 webserver.cc:460] Webserver started at http://127.22.113.1:40531/ using document root <none> and password file <none>
I20260619 06:53:41.529660 31734 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:41.529732 31734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:41.533761 31734 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:53:41.535920 31768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.537243 31734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:53:41.537334 31734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "79e3140c251c492ba048e7f6fcf1a5d5"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a8a024c40ebd8d163c0399e5bd379be8"
server_key_iv: "0fcf67597735b614265a7eec3ae9b766"
server_key_version: "encryptionkey@0"
I20260619 06:53:41.537905 31734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:41.553170 31734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:41.557602 31734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:41.557821 31734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:41.558557 31734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:41.559696 31734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:41.559757 31734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.559813 31734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:41.559835 31734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.572729 31734 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:35857
I20260619 06:53:41.572750 31881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:35857 every 8 connection(s)
I20260619 06:53:41.574184 31734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:41.580353 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31734
I20260619 06:53:41.580878 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:35379
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:32801
--webserver_port=43723
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:53:41.590848 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.576789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:41259 (local address 127.22.113.62:32801)
0619 06:53:41.577346 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.577375 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.577424 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:53:41.578992 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.579008 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.579521 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.579876 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.581508 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.582599 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.584521 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.585138 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.585304 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.589886 (+ 4582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:41259: BlockingRecv error: recv got EOF from 127.22.113.1:41259 (error 108)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":173,"threads_started":1}
I20260619 06:53:41.590857 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.576672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:41259)
0619 06:53:41.577199 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.577252 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.578498 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.579977 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.580038 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.580581 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.581289 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.581312 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.582767 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.582772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.584348 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.584363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.585337 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.585343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.585586 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.586561 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.586589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.589649 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":195,"threads_started":1}
W20260619 06:53:41.591732 31882 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:32801 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.113.62:32801: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:41.599725 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.592324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:39313)
0619 06:53:41.592545 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.592566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.592652 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.592995 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.592998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.593148 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.593292 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.593298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.594466 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.594470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.595624 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.595634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.596473 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.596476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.596609 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.597354 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.597390 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.599573 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260619 06:53:41.599957 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.592488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:39313 (local address 127.22.113.62:32801)
0619 06:53:41.592699 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.592704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.592723 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:53:41.592776 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.592780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.592839 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.592940 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.593415 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.594279 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.595776 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.596189 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.596317 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.599766 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:39313: BlockingRecv error: recv got EOF from 127.22.113.1:39313 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:41.606310 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.600288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:47325)
0619 06:53:41.600664 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.600681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.600793 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.601086 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.601089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.601249 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.601355 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.601361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.602112 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.602115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.603003 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.603015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.603586 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.603589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.603678 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.604155 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.604168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.606126 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":117,"threads_started":1}
W20260619 06:53:41.606355 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.600485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:47325 (local address 127.22.113.62:32801)
0619 06:53:41.600617 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.600620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.600739 (+ 119us) server_negotiation.cc:407] Connection header received
0619 06:53:41.600904 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.600906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.600951 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.601033 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.601468 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.601999 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.603165 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.603469 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.603565 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.606246 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:47325: BlockingRecv error: recv got EOF from 127.22.113.1:47325 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:53:41.606674 31882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:53:41.699725 31886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:41.699985 31886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:41.700026 31886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:41.703995 31886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:41.704056 31886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:41.704113 31886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:41.709334 31886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.2.keytab
--principal=oryx/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:35379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=43723
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:41.710384 31886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:41.711403 31886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:41.721729 31893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.721729 31895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.721861 31892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.722035 31886 server_base.cc:1034] running on GCE node
I20260619 06:53:41.722573 31886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:41.723215 31886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:41.724447 31886 hybrid_clock.cc:648] HybridClock initialized: now 1781852021724423 us; error 43 us; skew 500 ppm
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852021, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:41.727412 31886 init.cc:377] Logged in from keytab as oryx/127.22.113.2@KRBTEST.COM (short username oryx)
I20260619 06:53:41.728389 31886 webserver.cc:460] Webserver started at http://127.22.113.2:43723/ using document root <none> and password file <none>
I20260619 06:53:41.728979 31886 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:41.729082 31886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:41.733047 31886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:53:41.735484 31902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.736922 31886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:41.737089 31886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "21ac06c215444043a9eabbec12fe8242"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a54add6ef9059ce060ad985723019595"
server_key_iv: "baf7e3b02069d87c568f6ed8b07f1ba5"
server_key_version: "encryptionkey@0"
I20260619 06:53:41.737684 31886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:41.749322 31886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:41.753973 31886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:41.754207 31886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:41.755043 31886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:41.756184 31886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:41.756255 31886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.756317 31886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:41.756348 31886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.767417 31886 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:35379
I20260619 06:53:41.767482 32015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:35379 every 8 connection(s)
I20260619 06:53:41.768453 31886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:41.768782 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 31886
I20260619 06:53:41.769415 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:36589
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:32801
--webserver_port=39773
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:53:41.784256 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.770672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:38013 (local address 127.22.113.62:32801)
0619 06:53:41.770898 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.770907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.771789 (+ 882us) server_negotiation.cc:407] Connection header received
0619 06:53:41.774120 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.774124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.774189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.774286 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.776470 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.777028 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.779349 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.779590 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.779675 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.784122 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:38013: BlockingRecv error: recv got EOF from 127.22.113.2:38013 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:53:41.784951 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.770989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:38013)
0619 06:53:41.771595 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.771653 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.773766 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.774570 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.774585 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.775310 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.776273 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.776300 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.777185 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.777192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.779127 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.779143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.779785 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.779793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.780097 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.780864 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.780895 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.784010 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":192,"threads_started":1}
W20260619 06:53:41.785482 32016 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:32801 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.113.62:32801: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:41.792721 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.786056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:44471)
0619 06:53:41.786209 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.786228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.786374 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.786846 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.786850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.787057 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.787185 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.787193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.788195 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.788199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.789258 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.789269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.789725 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.789728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.789844 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.790374 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.790391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.792575 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:53:41.792836 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.786156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:44471 (local address 127.22.113.62:32801)
0619 06:53:41.786275 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.786279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.786291 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:41.786518 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.786522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.786591 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.786687 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.787363 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.788079 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.789388 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.789607 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.789689 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.792720 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:44471: BlockingRecv error: recv got EOF from 127.22.113.2:44471 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:41.799032 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.793471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:57629)
0619 06:53:41.793563 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.793578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.793681 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.793957 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.793960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.794123 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.794207 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.794213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.794921 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.794926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.795580 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.795588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.796109 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.796113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.796203 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.796786 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.796797 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.798865 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 06:53:41.799047 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.793601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57629 (local address 127.22.113.62:32801)
0619 06:53:41.793759 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.793763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.793775 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:53:41.793808 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.793811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.793855 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.793940 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.794319 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.794815 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.795695 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.796008 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.796106 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.798960 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:57629: BlockingRecv error: recv got EOF from 127.22.113.2:57629 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 06:53:41.799499 32016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:53:41.901605 32018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:41.901916 32018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:41.901991 32018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:41.906149 32018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:41.906243 32018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:41.906342 32018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:41.911963 32018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/krb5kdc/oryx_127.22.113.3.keytab
--principal=oryx/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:36589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=39773
--tserver_master_addrs=127.22.113.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:41.913286 32018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:41.914510 32018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:41.926097 32024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.926205 32018 server_base.cc:1034] running on GCE node
W20260619 06:53:41.926214 32025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:41.926231 32027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:41.926911 32018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:41.927784 32018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:41.928963 32018 hybrid_clock.cc:648] HybridClock initialized: now 1781852021928939 us; error 39 us; skew 500 ppm
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852021, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:53:41.932238 32018 init.cc:377] Logged in from keytab as oryx/127.22.113.3@KRBTEST.COM (short username oryx)
I20260619 06:53:41.933349 32018 webserver.cc:460] Webserver started at http://127.22.113.3:39773/ using document root <none> and password file <none>
I20260619 06:53:41.934026 32018 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:41.934136 32018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:41.938686 32018 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:41.941227 32034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.942903 32018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:41.943006 32018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "2584e98fb19e43059a91bb369cfdc89e"
format_stamp: "Formatted at 2026-06-19 06:53:40 on dist-test-slave-g8zg"
server_key: "a921fbfac348a228b681c03eca1239a1"
server_key_iv: "f1db29c2e33d0f2fa19b6dd17161003d"
server_key_version: "encryptionkey@0"
I20260619 06:53:41.943753 32018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:41.963078 32018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:41.966712 32018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:41.966899 32018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:41.967594 32018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:41.968590 32018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:41.968647 32018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.968693 32018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:41.968708 32018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:41.980120 32018 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:36589
I20260619 06:53:41.980238 32147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:36589 every 8 connection(s)
I20260619 06:53:41.981365 32018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:41.988361 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32018
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:53:41.994715 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.983535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46095 (local address 127.22.113.62:32801)
0619 06:53:41.983706 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.983710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.984764 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 06:53:41.986172 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.986176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.986232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.986313 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.987567 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.988144 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.989887 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.990254 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.990362 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.994590 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:46095: BlockingRecv error: recv got EOF from 127.22.113.3:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:53:41.995450 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.984049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:46095)
0619 06:53:41.984593 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.984641 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.985949 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.986488 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.986496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.986879 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.987429 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.987444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.988354 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.988359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.989696 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.989712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.990434 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.990440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.990669 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.991437 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:41.991454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:41.994457 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
W20260619 06:53:41.995987 32148 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:32801 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.113.62:32801: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
I20260619 06:53:42.000075 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.989572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.22.113.62:32801)
0619 06:53:41.989923 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.989927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.989981 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:53:41.990125 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.990128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.990172 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.990287 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.991331 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.991837 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.992883 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.993144 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.993270 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:41.995088 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:41.995094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:41.995112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:53:41.995141 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:41.997731 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:41.998214 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:41.998219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:41.998224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:41.998334 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:53:41.998621 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:53:41.998626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:53:41.998628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:53:41.999014 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:41.999240 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:41.999569 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 06:53:41.999828 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:41 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:42.002451 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:41.996571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:39219)
0619 06:53:41.996968 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:41.996988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:41.997097 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:41.997343 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:41.997346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:41.997566 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:41.997666 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.997672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.998512 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.998515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.999077 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:41.999084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.999598 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.999602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:41.999700 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.000213 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:42.000233 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:42.002346 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321}
W20260619 06:53:42.002925 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:41.996708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:39219 (local address 127.22.113.62:32801)
0619 06:53:41.996887 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:41.996893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:41.997052 (+ 159us) server_negotiation.cc:407] Connection header received
0619 06:53:41.997179 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:41.997183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:41.997228 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:41.997328 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:41.997776 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.998405 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.999196 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:41.999479 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:41.999583 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.002804 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:39219: BlockingRecv error: recv got EOF from 127.22.113.3:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:42.008080 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:42.003099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:42569)
0619 06:53:42.003341 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:42.003359 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:42.003558 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:42.003804 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:42.003806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:42.003929 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:42.003995 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.004001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.004728 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.004731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.005231 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.005238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.005634 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.005637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.005691 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.006085 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:42.006095 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:42.007927 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172}
W20260619 06:53:42.008136 31885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:42.003425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:42569 (local address 127.22.113.62:32801)
0619 06:53:42.003619 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:42.003622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:42.003633 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:53:42.003663 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:42.003665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:42.003699 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:42.003767 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:42.004106 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.004608 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.005336 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.005539 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.005613 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.008034 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:42569: BlockingRecv error: recv got EOF from 127.22.113.3:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260619 06:53:42.008392 32148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:42.618747 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:42.607457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:42065)
0619 06:53:42.607793 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:42.607808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:42.607928 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:42.608240 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:42.608243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:42.608459 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:42.608568 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.608575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.609367 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.609379 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.609957 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.609965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.610464 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.610468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.610553 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.616633 (+ 6080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:42.616648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:42.618563 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
W20260619 06:53:42.618821 32154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:42.607496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:42065 (local address 127.22.113.62:32801)
0619 06:53:42.607803 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:42.607808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:42.607867 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:53:42.608052 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:42.608056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:42.608111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:42.608222 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:42.608690 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.609249 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.610091 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.610346 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.610464 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.618712 (+ 8248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:42065: BlockingRecv error: recv got EOF from 127.22.113.1:42065 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:42 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:42.805917 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:42.800181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:52307)
0619 06:53:42.800483 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:42.800505 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:42.800639 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:42.800918 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:42.800921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:42.801138 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:42.801226 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.801231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.801981 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.801987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.802558 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:42.802565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.803008 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.803012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:42.803094 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.803697 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:42.803709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:42.805772 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
W20260619 06:53:42.806245 32154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:42.800254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:52307 (local address 127.22.113.62:32801)
0619 06:53:42.800454 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:42.800459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:42.800583 (+ 124us) server_negotiation.cc:407] Connection header received
0619 06:53:42.800755 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:42.800759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:42.800810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:42.800899 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:42.801343 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.801875 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.802684 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:42.802901 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:42.803005 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:42.805927 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:52307: BlockingRecv error: recv got EOF from 127.22.113.2:52307 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:43.014279 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:43.009029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:49231)
0619 06:53:43.009278 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:43.009291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:43.009387 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:43.009615 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:43.009617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:43.009788 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:43.009880 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.009885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.010619 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.010622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.011195 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.011203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.011665 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.011668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.011727 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.012292 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:43.012303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:43.014152 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
W20260619 06:53:43.014483 32154 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:43.009094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:49231 (local address 127.22.113.62:32801)
0619 06:53:43.009235 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:43.009240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:43.009360 (+ 120us) server_negotiation.cc:407] Connection header received
0619 06:53:43.009460 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:43.009462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:43.009506 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:43.009575 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:43.009972 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.010504 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.011330 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.011557 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.011649 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.014313 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:49231: BlockingRecv error: recv got EOF from 127.22.113.3:49231 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:43.625582 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:43.619671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:40575)
0619 06:53:43.620018 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:43.620036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:43.620212 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:43.620571 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:43.620574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:43.620801 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:43.620927 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.620932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.621724 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.621727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.622347 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.622355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.622816 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.622819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.622881 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.623448 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:43.623460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:43.625433 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
W20260619 06:53:43.625698 32158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:43.619798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40575 (local address 127.22.113.62:32801)
0619 06:53:43.620037 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:43.620041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:43.620147 (+ 106us) server_negotiation.cc:407] Connection header received
0619 06:53:43.620362 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:43.620364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:43.620425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:43.620514 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:43.621068 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.621616 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.622470 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.622705 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.622792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.625575 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:40575: BlockingRecv error: recv got EOF from 127.22.113.1:40575 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":59,"threads_started":1}
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:43 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:43.812827 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:43.806959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:48017)
0619 06:53:43.807345 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:43.807365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:43.807500 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:43.807812 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:43.807816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:43.808073 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:43.808184 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.808190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.809000 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.809007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.809584 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:43.809592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.810043 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.810046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:43.810131 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.810677 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:43.810689 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:43.812602 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
W20260619 06:53:43.812847 32158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:43.807076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:48017 (local address 127.22.113.62:32801)
0619 06:53:43.807267 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:43.807273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:43.807483 (+ 210us) server_negotiation.cc:407] Connection header received
0619 06:53:43.807611 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:43.807616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:43.807678 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:43.807786 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:43.808311 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.808890 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.809702 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:43.809927 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:43.810030 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:43.812735 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:48017: BlockingRecv error: recv got EOF from 127.22.113.2:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:44.021831 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:44.015421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:46761)
0619 06:53:44.015750 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:44.015763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:44.015852 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:44.016172 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:44.016174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:44.016349 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:44.016450 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.016455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.017331 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.017335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.017977 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.017986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.018482 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.018485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.018559 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.019208 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:44.019220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:44.021617 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":123,"threads_started":1}
W20260619 06:53:44.022090 32158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:44.015630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46761 (local address 127.22.113.62:32801)
0619 06:53:44.015818 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:44.015822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:44.015849 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:53:44.015963 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:44.015966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:44.016027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:44.016133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:44.016599 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.017184 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.018121 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.018355 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.018444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.021949 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:46761: BlockingRecv error: recv got EOF from 127.22.113.3:46761 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:44.633239 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:44.626845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:51993)
0619 06:53:44.627160 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:44.627174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:44.627377 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:44.627732 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:44.627734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:44.627953 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:44.628110 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.628116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.628963 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.628965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.629601 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.629609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.630050 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.630052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.630112 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.630726 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:44.630738 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:44.633042 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
W20260619 06:53:44.633337 32162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:44.626905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:51993 (local address 127.22.113.62:32801)
0619 06:53:44.627207 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:44.627211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:44.627234 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:53:44.627524 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:44.627526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:44.627581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:44.627683 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:44.628274 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.628823 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.629716 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.629940 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.630029 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.633188 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:51993: BlockingRecv error: recv got EOF from 127.22.113.1:51993 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":111,"threads_started":1}
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:44 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:44.819599 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:44.813733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:35153)
0619 06:53:44.814149 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:44.814171 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:44.814311 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:44.814619 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:44.814622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:44.814854 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:44.814949 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.814955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.815805 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.815811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.816383 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:44.816390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.816860 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.816863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:44.816945 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.817567 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:44.817578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:44.819414 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":110,"threads_started":1}
W20260619 06:53:44.819649 32162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:44.813874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:35153 (local address 127.22.113.62:32801)
0619 06:53:44.814098 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:44.814104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:44.814255 (+ 151us) server_negotiation.cc:407] Connection header received
0619 06:53:44.814425 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:44.814430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:44.814501 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:44.814593 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:44.815068 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.815694 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.816504 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:44.816732 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:44.816849 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:44.819534 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:35153: BlockingRecv error: recv got EOF from 127.22.113.2:35153 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:45.029433 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:45.023214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:54515)
0619 06:53:45.023629 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:45.023649 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:45.023803 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:45.024116 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:45.024118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:45.024403 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:45.024557 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.024564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.025362 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.025368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.025965 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.025972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.026436 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.026439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.026526 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.027162 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:45.027174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:45.029295 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":120,"threads_started":1}
W20260619 06:53:45.029585 32162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:45.023349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:54515 (local address 127.22.113.62:32801)
0619 06:53:45.023533 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:45.023538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:45.023760 (+ 222us) server_negotiation.cc:407] Connection header received
0619 06:53:45.023935 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:45.023939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:45.024002 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:45.024093 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:45.024676 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.025250 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.026089 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.026317 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.026430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.029439 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:54515: BlockingRecv error: recv got EOF from 127.22.113.3:54515 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:45.640650 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:45.634232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:54115)
0619 06:53:45.634584 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:45.634605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:45.634753 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:45.635098 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:45.635101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:45.635583 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:45.635714 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.635720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.636774 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.636780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.637423 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.637439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.637912 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.637916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.637993 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.638526 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:45.638538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:45.640513 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
W20260619 06:53:45.640936 32166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:45.634405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:54115 (local address 127.22.113.62:32801)
0619 06:53:45.634803 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:45.634808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:45.634824 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:45.634877 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:45.634879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:45.634925 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:45.635023 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:45.635867 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.636654 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.637545 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.637792 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.637902 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.640736 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:54115: BlockingRecv error: recv got EOF from 127.22.113.1:54115 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":148,"threads_started":1}
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:45 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:45.826838 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:45.820682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:52751)
0619 06:53:45.821010 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:45.821027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:45.821168 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:45.821488 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:45.821490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:45.821675 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:45.821775 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.821780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.822576 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.822580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.823175 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:45.823184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.823752 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.823756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:45.823851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.824511 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:45.824548 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:45.826716 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
W20260619 06:53:45.827003 32166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:45.820791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:52751 (local address 127.22.113.62:32801)
0619 06:53:45.821009 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:45.821013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:45.821107 (+ 94us) server_negotiation.cc:407] Connection header received
0619 06:53:45.821266 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:45.821268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:45.821324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:45.821405 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:45.821874 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.822459 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.823336 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:45.823618 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:45.823729 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:45.826853 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:52751: BlockingRecv error: recv got EOF from 127.22.113.2:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:46.036863 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:46.030435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:48189)
0619 06:53:46.030836 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:46.030853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:46.031014 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:46.031379 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:46.031382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:46.031596 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:46.031690 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.031695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.032473 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.032476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.033097 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.033113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.033570 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.033574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.033650 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.034365 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:46.034377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:46.036658 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
W20260619 06:53:46.036917 32166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:46.030569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:48189 (local address 127.22.113.62:32801)
0619 06:53:46.030780 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:46.030784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:46.030949 (+ 165us) server_negotiation.cc:407] Connection header received
0619 06:53:46.031143 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:46.031148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:46.031203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:46.031354 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:46.031826 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.032357 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.033224 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.033453 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.033581 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.036801 (+ 3220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:48189: BlockingRecv error: recv got EOF from 127.22.113.3:48189 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:46.648337 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:46.641697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:45853)
0619 06:53:46.642089 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:46.642110 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:46.642263 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:46.642583 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:46.642586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:46.642852 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:46.642976 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.642982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.643867 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.643873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.644461 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.644469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.644950 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.644954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.645039 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.645713 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:46.645728 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:46.648206 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
W20260619 06:53:46.648559 32170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:46.641781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:45853 (local address 127.22.113.62:32801)
0619 06:53:46.642113 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:46.642118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:46.642199 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:53:46.642365 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:46.642369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:46.642428 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:46.642560 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:46.643096 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.643759 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.644593 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.644834 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.644938 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.648349 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:45853: BlockingRecv error: recv got EOF from 127.22.113.1:45853 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":62,"threads_started":1}
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:46 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:46.833846 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:46.827755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:40789)
0619 06:53:46.828138 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:46.828156 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:46.828316 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:46.828632 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:46.828635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:46.828891 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:46.829014 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.829020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.829835 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.829841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.830440 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:46.830449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.830899 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.830903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:46.830994 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.831599 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:46.831611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:46.833718 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
W20260619 06:53:46.834004 32170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:46.827941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:40789 (local address 127.22.113.62:32801)
0619 06:53:46.828126 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:46.828131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:46.828247 (+ 116us) server_negotiation.cc:407] Connection header received
0619 06:53:46.828424 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:46.828429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:46.828512 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:46.828605 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:46.829128 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.829727 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.830557 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:46.830776 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:46.830879 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:46.833858 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:40789: BlockingRecv error: recv got EOF from 127.22.113.2:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:47.043629 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:47.037892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:41877)
0619 06:53:47.038210 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:47.038225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:47.038333 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:47.038585 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:47.038587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:47.038767 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:47.038859 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.038863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.039681 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.039684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.040272 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.040280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.040749 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.040754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.040837 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.041439 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:47.041450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:47.043506 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260619 06:53:47.043846 32170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:47.037974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:41877 (local address 127.22.113.62:32801)
0619 06:53:47.038134 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:47.038138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:47.038328 (+ 190us) server_negotiation.cc:407] Connection header received
0619 06:53:47.038412 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:47.038415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:47.038468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:47.038547 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:47.038962 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.039583 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.040374 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.040600 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.040681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.043664 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:41877: BlockingRecv error: recv got EOF from 127.22.113.3:41877 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:47.658808 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:47.649206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:33425)
0619 06:53:47.649520 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:47.649534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:47.649637 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:47.649910 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:47.649912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:47.650126 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:47.650235 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.650240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.651073 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.651075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.651703 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.651712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.652135 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.652137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.652196 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.656372 (+ 4176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:47.656391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:47.658625 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20260619 06:53:47.659001 32174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:47.649294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:33425 (local address 127.22.113.62:32801)
0619 06:53:47.649578 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:47.649582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:47.649596 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:53:47.649723 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:47.649725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:47.649775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:47.649870 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:47.650381 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.650970 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.651816 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.652040 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.652121 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.658817 (+ 6696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:33425: BlockingRecv error: recv got EOF from 127.22.113.1:33425 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:47 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:47.840502 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:47.834681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:35693)
0619 06:53:47.834988 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:47.835003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:47.835131 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:47.835426 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:47.835428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:47.835658 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:47.835772 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.835778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.836550 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.836553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.837180 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:47.837188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.837662 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.837665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:47.837731 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.838277 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:47.838289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:47.840333 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260619 06:53:47.840619 32174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:47.834825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:35693 (local address 127.22.113.62:32801)
0619 06:53:47.835021 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:47.835025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:47.835111 (+ 86us) server_negotiation.cc:407] Connection header received
0619 06:53:47.835219 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:47.835221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:47.835295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:47.835379 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:47.835876 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.836449 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.837293 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:47.837545 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:47.837628 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:47.840479 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:35693: BlockingRecv error: recv got EOF from 127.22.113.2:35693 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:48.050158 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:48.044490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:52623)
0619 06:53:48.044778 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:48.044792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:48.044905 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:48.045143 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:48.045145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:48.045354 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:48.045463 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.045469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.046204 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.046206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.046777 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.046803 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.047354 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.047359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.047426 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.048013 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:48.048032 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:48.049979 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
W20260619 06:53:48.050433 32174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:48.044558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:52623 (local address 127.22.113.62:32801)
0619 06:53:48.044706 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:48.044710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:48.044864 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:53:48.044990 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:48.044992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:48.045043 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:48.045118 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:48.045557 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.046118 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.047017 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.047229 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.047332 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.050137 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:52623: BlockingRecv error: recv got EOF from 127.22.113.3:52623 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:48.665755 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:48.659766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:42415)
0619 06:53:48.660141 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:48.660156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:48.660263 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:48.660574 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:48.660576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:48.660787 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:48.660906 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.660911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.661730 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.661732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.662328 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.662335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.662826 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.662829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.662890 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.663520 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:48.663532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:48.665521 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":105,"threads_started":1}
W20260619 06:53:48.665758 32178 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:48.659962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:42415 (local address 127.22.113.62:32801)
0619 06:53:48.660230 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:48.660234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:48.660252 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:48.660365 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:48.660367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:48.660416 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:48.660540 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:48.661056 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.661625 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.662458 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.662714 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.662813 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.665638 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:42415: BlockingRecv error: recv got EOF from 127.22.113.1:42415 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:48 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:48.848120 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:48.841512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:36717)
0619 06:53:48.841882 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:48.841900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:48.842030 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:48.842378 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:48.842381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:48.842658 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:48.842786 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.842793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.843901 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.843906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.844508 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:48.844517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.845096 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.845099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:48.845189 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.845756 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:48.845767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:48.847936 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
W20260619 06:53:48.848271 32178 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:48.841653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:36717 (local address 127.22.113.62:32801)
0619 06:53:48.841886 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:48.841890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:48.841993 (+ 103us) server_negotiation.cc:407] Connection header received
0619 06:53:48.842162 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:48.842168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:48.842239 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:48.842317 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:48.842895 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.843780 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.844634 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:48.844978 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:48.845091 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:48.848129 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:36717: BlockingRecv error: recv got EOF from 127.22.113.2:36717 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:49.056911 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:49.050991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:53631)
0619 06:53:49.051429 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:49.051445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:49.051577 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:49.051889 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:49.051891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:49.052074 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:49.052184 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.052189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.052976 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.052979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.053562 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.053570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.054028 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.054030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.054093 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.054672 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:49.054684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:49.056715 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":139,"threads_started":1}
W20260619 06:53:49.057014 32178 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:49.051095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:53631 (local address 127.22.113.62:32801)
0619 06:53:49.051330 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:49.051338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:49.051546 (+ 208us) server_negotiation.cc:407] Connection header received
0619 06:53:49.051708 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:49.051710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:49.051771 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:49.051847 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:49.052286 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.052852 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.053678 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.053917 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.054000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.056883 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:53631: BlockingRecv error: recv got EOF from 127.22.113.3:53631 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:49.672796 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:49.666731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:36269)
0619 06:53:49.667067 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:49.667087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:49.667215 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:49.667595 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:49.667599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:49.667842 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:49.667949 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.667955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.668951 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.668953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.669557 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.669565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.670021 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.670025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.670094 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.670650 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:49.670662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:49.672652 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":126,"threads_started":1}
W20260619 06:53:49.672971 32182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:49.666867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:36269 (local address 127.22.113.62:32801)
0619 06:53:49.667165 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:49.667169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:49.667184 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:53:49.667354 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:49.667357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:49.667418 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:49.667542 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:49.668110 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.668841 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.669665 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.669892 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.669979 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.672810 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:36269: BlockingRecv error: recv got EOF from 127.22.113.1:36269 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:49 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:49.854921 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:49.849097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:59479)
0619 06:53:49.849421 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:49.849435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:49.849549 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:49.849830 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:49.849832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:49.850019 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:49.850125 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.850132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.851107 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.851110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.851794 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:49.851802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.852229 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.852232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:49.852296 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.852849 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:49.852861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:49.854795 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":113,"threads_started":1}
W20260619 06:53:49.855134 32182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:49.849163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:59479 (local address 127.22.113.62:32801)
0619 06:53:49.849335 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:49.849340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:49.849547 (+ 207us) server_negotiation.cc:407] Connection header received
0619 06:53:49.849651 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:49.849653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:49.849713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:49.849785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:49.850249 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.850984 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.851902 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:49.852124 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:49.852207 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:49.854957 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:59479: BlockingRecv error: recv got EOF from 127.22.113.2:59479 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:50.063984 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:50.057796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:45647)
0619 06:53:50.058079 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:50.058093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:50.058212 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:50.058521 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:50.058524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:50.058729 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:50.058832 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.058838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.059978 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.059982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.060594 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.060602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.061065 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.061068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.061125 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.061720 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:50.061731 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:50.063869 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
W20260619 06:53:50.064225 32182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:50.057908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:45647 (local address 127.22.113.62:32801)
0619 06:53:50.058070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:50.058074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:50.058187 (+ 113us) server_negotiation.cc:407] Connection header received
0619 06:53:50.058307 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:50.058310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:50.058375 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:50.058470 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:50.058951 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.059852 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.060714 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.060962 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.061056 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.064051 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:45647: BlockingRecv error: recv got EOF from 127.22.113.3:45647 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:50.679656 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:50.673833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:51345)
0619 06:53:50.674129 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:50.674143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:50.674231 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:50.674576 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:50.674579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:50.674762 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:50.674868 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.674873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.675749 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.675751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.676350 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.676358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.676829 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.676832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.676895 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.677475 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:50.677487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:50.679500 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
W20260619 06:53:50.679960 32186 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:50.673991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:51345 (local address 127.22.113.62:32801)
0619 06:53:50.674271 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:50.674275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:50.674289 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:53:50.674334 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:50.674337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:50.674387 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:50.674510 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:50.674989 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.675626 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.676479 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.676721 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.676809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.679653 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:51345: BlockingRecv error: recv got EOF from 127.22.113.1:51345 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:50 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:50.862408 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:50.855910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:42463)
0619 06:53:50.856308 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:50.856327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:50.856552 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:50.856848 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:50.856852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:50.857103 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:50.857207 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.857213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.858003 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.858009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.858602 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:50.858610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.859070 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.859073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:50.859165 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.859764 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:50.859779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:50.862184 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
W20260619 06:53:50.862432 32186 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:50.856107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:42463 (local address 127.22.113.62:32801)
0619 06:53:50.856312 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:50.856320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:50.856472 (+ 152us) server_negotiation.cc:407] Connection header received
0619 06:53:50.856668 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:50.856672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:50.856730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:50.856823 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:50.857330 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.857887 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.858719 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:50.858956 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:50.859064 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:50.862318 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:42463: BlockingRecv error: recv got EOF from 127.22.113.2:42463 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:51.070647 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:51.064834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:42757)
0619 06:53:51.065194 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:51.065210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:51.065364 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:51.065685 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:51.065687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:51.065873 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:51.065978 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.065983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.066756 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.066758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.067341 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.067349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.067774 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.067777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.067834 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.068368 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:51.068383 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:51.070467 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":140,"threads_started":1}
W20260619 06:53:51.070766 32186 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:51.064921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:42757 (local address 127.22.113.62:32801)
0619 06:53:51.065090 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:51.065095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:51.065299 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:53:51.065499 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:51.065501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:51.065561 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:51.065637 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:51.066097 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.066657 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.067449 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.067672 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.067757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.070617 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:42757: BlockingRecv error: recv got EOF from 127.22.113.3:42757 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:51.686520 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:51.680584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:60757)
0619 06:53:51.680890 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:51.680911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:51.681060 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:51.681340 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:51.681342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:51.681552 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:51.681673 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.681678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.682445 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.682447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.683044 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.683051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.683524 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.683527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.683594 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.684235 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:51.684247 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:51.686335 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
W20260619 06:53:51.686578 32190 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:51.680754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:60757 (local address 127.22.113.62:32801)
0619 06:53:51.681093 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:51.681096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:51.681112 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:51.681149 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:51.681151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:51.681205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:51.681291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:51.681780 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.682326 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.683157 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.683413 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.683500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.686453 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:60757: BlockingRecv error: recv got EOF from 127.22.113.1:60757 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":161,"threads_started":1}
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:51 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:51.869529 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:51.863411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:43949)
0619 06:53:51.863739 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:51.863760 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:51.863878 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:51.864263 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:51.864265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:51.864479 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:51.864595 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.864600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.865361 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.865365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.865979 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:51.865988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.866473 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.866476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:51.866538 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.867157 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:51.867170 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:51.869336 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
W20260619 06:53:51.869769 32190 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:51.863880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:43949 (local address 127.22.113.62:32801)
0619 06:53:51.864027 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:51.864031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:51.864047 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:53:51.864094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:51.864096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:51.864142 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:51.864221 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:51.864704 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.865224 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.866134 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:51.866356 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:51.866455 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:51.869493 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:43949: BlockingRecv error: recv got EOF from 127.22.113.2:43949 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:52.077742 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:52.071584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:59999)
0619 06:53:52.071960 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:52.071978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:52.072134 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:52.072412 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:52.072414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:52.072664 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:52.072773 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.072779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.073585 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.073591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.074171 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.074179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.074640 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.074643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.074723 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.075359 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:52.075370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:52.077574 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":145,"threads_started":1}
W20260619 06:53:52.077850 32190 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:52.071778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:59999 (local address 127.22.113.62:32801)
0619 06:53:52.071960 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:52.071974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:52.072064 (+ 90us) server_negotiation.cc:407] Connection header received
0619 06:53:52.072236 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:52.072240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:52.072298 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:52.072388 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:52.072889 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.073474 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.074285 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.074529 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.074638 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.077713 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:59999: BlockingRecv error: recv got EOF from 127.22.113.3:59999 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:52.698203 32193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:52.687718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:44725)
0619 06:53:52.688107 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:52.688123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:52.688238 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:52.688553 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:52.688556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:52.688748 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:52.688878 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.688883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.689933 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.689937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.690798 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.690807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.691340 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.691344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.691418 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.695686 (+ 4268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:52.695698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:52.698078 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
W20260619 06:53:52.698473 32194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:52.687821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44725 (local address 127.22.113.62:32801)
0619 06:53:52.688188 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:52.688192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:52.688210 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:52.688347 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:52.688349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:52.688405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:52.688504 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:52.689020 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.689780 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.690906 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.691161 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.691297 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.698275 (+ 6978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:44725: BlockingRecv error: recv got EOF from 127.22.113.1:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:52 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:52.876456 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:52.870425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:60589)
0619 06:53:52.870727 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:52.870741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:52.870880 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:52.871169 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:52.871172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:52.871463 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:52.871583 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.871589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.872377 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.872379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.873003 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:52.873011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.873465 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.873467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:52.873529 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.874054 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:52.874066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:52.876270 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
W20260619 06:53:52.876497 32194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:52.870554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:60589 (local address 127.22.113.62:32801)
0619 06:53:52.870730 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:52.870734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:52.870856 (+ 122us) server_negotiation.cc:407] Connection header received
0619 06:53:52.870980 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:52.870983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:52.871044 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:52.871128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:52.871719 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.872273 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.873105 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:52.873343 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:52.873424 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:52.876388 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:60589: BlockingRecv error: recv got EOF from 127.22.113.2:60589 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:53.086294 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:53.078550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:60181)
0619 06:53:53.079160 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:53.079182 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:53.079421 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:53.079856 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:53.079860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:53.080169 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:53.080335 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.080344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.081510 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.081515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.082433 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.082446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.083076 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.083082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.083204 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.083982 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:53.083996 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:53.086068 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":135,"threads_started":1}
W20260619 06:53:53.086356 32194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:53.078676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:60181 (local address 127.22.113.62:32801)
0619 06:53:53.078911 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:53.078918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:53.079326 (+ 408us) server_negotiation.cc:407] Connection header received
0619 06:53:53.079600 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:53.079606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:53.079673 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:53.079778 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:53.080496 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.081316 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.082565 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.082911 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.083029 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.086234 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:60181: BlockingRecv error: recv got EOF from 127.22.113.3:60181 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:53.705214 32198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:53.699178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:48805)
0619 06:53:53.699557 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:53.699574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:53.699689 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:53.699973 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:53.699976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:53.700177 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:53.700280 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.700286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.701171 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.701173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.701780 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.701794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.702283 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.702285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.702345 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.702949 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:53.702961 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:53.705069 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":166,"threads_started":1}
W20260619 06:53:53.705458 32197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:53.699211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:48805 (local address 127.22.113.62:32801)
0619 06:53:53.699562 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:53.699566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:53.699664 (+ 98us) server_negotiation.cc:407] Connection header received
0619 06:53:53.699777 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:53.699779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:53.699835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:53.699955 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:53.700450 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.701044 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.701931 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.702172 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.702263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.705279 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:48805: BlockingRecv error: recv got EOF from 127.22.113.1:48805 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:53 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:53.883462 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:53.877318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:36769)
0619 06:53:53.877672 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:53.877688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:53.877818 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:53.878078 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:53.878080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:53.878288 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:53.878408 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.878414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.879319 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.879322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.879927 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:53.879935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.880398 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.880400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:53.880482 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.881040 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:53.881052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:53.883207 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":75,"threads_started":1}
W20260619 06:53:53.883585 32197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:53.877453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:36769 (local address 127.22.113.62:32801)
0619 06:53:53.877673 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:53.877677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:53.877777 (+ 100us) server_negotiation.cc:407] Connection header received
0619 06:53:53.877910 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:53.877912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:53.877962 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:53.878040 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:53.878550 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.879168 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.880062 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:53.880288 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:53.880384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:53.883423 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:36769: BlockingRecv error: recv got EOF from 127.22.113.2:36769 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:54.093250 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:54.087216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:43625)
0619 06:53:54.087641 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:54.087654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:54.087780 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:54.088050 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:54.088053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:54.088273 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:54.088369 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.088374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.089186 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.089192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.089751 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.089758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.090230 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.090233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.090316 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.090888 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:54.090899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:54.093032 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":112,"threads_started":1}
W20260619 06:53:54.093263 32197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:54.087347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:43625 (local address 127.22.113.62:32801)
0619 06:53:54.087513 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:54.087519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:54.087732 (+ 213us) server_negotiation.cc:407] Connection header received
0619 06:53:54.087886 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:54.087889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:54.087941 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:54.088031 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:54.088488 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.089081 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.089867 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.090112 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.090212 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.093163 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:43625: BlockingRecv error: recv got EOF from 127.22.113.3:43625 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:54.712142 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:54.706160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:56059)
0619 06:53:54.706483 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:54.706504 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:54.706631 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:54.706895 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:54.706898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:54.707124 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:54.707267 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.707273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.708188 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.708191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.708815 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.708823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.709294 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.709296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.709358 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.709986 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:54.709998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:54.711926 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
W20260619 06:53:54.712240 32202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:54.706296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:56059 (local address 127.22.113.62:32801)
0619 06:53:54.706614 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:54.706620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:54.706640 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:53:54.706723 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:54.706725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:54.706771 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:54.706857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:54.707405 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.708065 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.708930 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.709185 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.709277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.712078 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:56059: BlockingRecv error: recv got EOF from 127.22.113.1:56059 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":138,"threads_started":1}
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:54 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:54.890416 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:54.884390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:35297)
0619 06:53:54.884765 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:54.884785 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:54.884958 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:54.885233 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:54.885235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:54.885440 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:54.885556 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.885561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.886282 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.886285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.886916 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:54.886924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.887434 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.887437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:54.887507 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.888147 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:54.888160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:54.890227 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
W20260619 06:53:54.890719 32202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:54.884639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:35297 (local address 127.22.113.62:32801)
0619 06:53:54.884787 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:54.884791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:54.884891 (+ 100us) server_negotiation.cc:407] Connection header received
0619 06:53:54.885058 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:54.885060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:54.885114 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:54.885189 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:54.885659 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.886183 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.887035 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:54.887289 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:54.887378 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:54.890411 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:35297: BlockingRecv error: recv got EOF from 127.22.113.2:35297 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:55.100211 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:55.094273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:55287)
0619 06:53:55.094635 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:55.094650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:55.094771 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:55.095064 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:55.095066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:55.095282 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:55.095384 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.095390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.096219 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.096222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.096934 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.096951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.097378 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.097381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.097442 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.098022 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:55.098033 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:55.100071 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
W20260619 06:53:55.100358 32202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:55.094370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:55287 (local address 127.22.113.62:32801)
0619 06:53:55.094543 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:55.094549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:55.094740 (+ 191us) server_negotiation.cc:407] Connection header received
0619 06:53:55.094864 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:55.094866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:55.094925 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:55.095011 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:55.095500 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.096112 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.097044 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.097268 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.097381 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.100211 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:55287: BlockingRecv error: recv got EOF from 127.22.113.3:55287 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:55.720703 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:55.713428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:36223)
0619 06:53:55.713774 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:55.713798 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:55.713948 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:55.714322 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:55.714324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:55.714549 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:55.714668 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.714673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.715506 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.715509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.716334 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.716344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.717125 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.717237 (+ 112us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.717345 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.717979 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:55.717991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:55.720451 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
W20260619 06:53:55.720839 32206 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:55.713593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:36223 (local address 127.22.113.62:32801)
0619 06:53:55.714060 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:55.714065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:55.714083 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:55.714128 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:55.714131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:55.714181 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:55.714280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:55.714800 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.715377 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.716481 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.716962 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.717130 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.720665 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:36223: BlockingRecv error: recv got EOF from 127.22.113.1:36223 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":130,"threads_started":1}
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:55 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:55.898337 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:55.891530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:55575)
0619 06:53:55.891877 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:55.891893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:55.892023 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:55.892354 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:55.892357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:55.892570 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:55.892689 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.892695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.893532 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.893535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.894172 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:55.894180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.894781 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.894788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:55.894890 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.895652 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:55.895668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:55.898180 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":141,"threads_started":1}
W20260619 06:53:55.898540 32206 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:55.891575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:55575 (local address 127.22.113.62:32801)
0619 06:53:55.891782 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:55.891785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:55.891987 (+ 202us) server_negotiation.cc:407] Connection header received
0619 06:53:55.892130 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:55.892132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:55.892184 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:55.892278 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:55.892834 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.893393 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.894321 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:55.894636 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:55.894725 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:55.898338 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:55575: BlockingRecv error: recv got EOF from 127.22.113.2:55575 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.3@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:56.106784 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:56.101197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.3:52219)
0619 06:53:56.101521 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:56.101536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:56.101652 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:56.101910 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:56.101911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:56.102118 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:56.102237 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.102242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.102968 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.102970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.103725 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.103734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.104139 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.104141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.104207 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.104772 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:56.104784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:56.106638 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":127,"threads_started":1}
W20260619 06:53:56.106910 32206 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:56.101252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:52219 (local address 127.22.113.62:32801)
0619 06:53:56.101415 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:56.101419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:56.101616 (+ 197us) server_negotiation.cc:407] Connection header received
0619 06:53:56.101741 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:56.101744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:56.101792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:56.101874 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:56.102346 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.102869 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.103827 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.104040 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.104121 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.106785 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.3:52219: BlockingRecv error: recv got EOF from 127.22.113.3:52219 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.1@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:56.727690 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:56.721697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.1:49059)
0619 06:53:56.721953 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:56.721966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:56.722053 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:56.722374 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:56.722376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:56.722548 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:56.722688 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.722693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.723537 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.723540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.724216 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.724226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.724699 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.724701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.724765 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.725409 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:56.725431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:56.727565 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
W20260619 06:53:56.727877 32210 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:56.721787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:49059 (local address 127.22.113.62:32801)
0619 06:53:56.722119 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:56.722124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:56.722142 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:53:56.722196 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:56.722198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:56.722244 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:56.722337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:56.722804 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.723418 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.724335 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.724573 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.724657 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.727713 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.1:49059: BlockingRecv error: recv got EOF from 127.22.113.1:49059 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:53:56 dist-test-slave-g8zg krb5kdc[31151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.113.2@KRBTEST.COM for oryx/127.22.113.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:53:56.905539 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:56.899460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32801 (local address 127.22.113.2:51595)
0619 06:53:56.899784 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:56.899799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:56.899955 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:56.900239 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:56.900242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:56.900449 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:56.900537 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.900543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.901302 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.901307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.901868 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:56.901875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.902323 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.902327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:56.902405 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.903039 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:53:56.903052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:56.905353 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.113.62:32801: Server oryx/127.22.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
W20260619 06:53:56.905589 32210 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:56.899435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:51595 (local address 127.22.113.62:32801)
0619 06:53:56.899588 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:56.899592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:56.899902 (+ 310us) server_negotiation.cc:407] Connection header received
0619 06:53:56.900058 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:56.900062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:56.900119 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:56.900219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:56.900654 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.901195 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.901993 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:56.902210 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:56.902314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:56.905472 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.113.2:51595: BlockingRecv error: recv got EOF from 127.22.113.2:51595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:53:56.991654 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31734
I20260619 06:53:56.998063 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31886
I20260619 06:53:57.004834 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32018
I20260619 06:53:57.011046 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 31664
2026-06-19T06:53:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18213 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:57Z Disabled control of system clock
I20260619 06:53:57.045990 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:46059
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:53:57.164234 32215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:57.164503 32215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:57.164537 32215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:57.168466 32215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:57.168522 32215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:57.168537 32215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:57.168553 32215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:57.168569 32215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:57.173557 32215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:46059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:57.174633 32215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:57.176086 32215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:57.185680 32221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.185806 32215 server_base.cc:1034] running on GCE node
W20260619 06:53:57.185720 32220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:57.185734 32223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.186611 32215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:57.187865 32215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:57.189038 32215 hybrid_clock.cc:648] HybridClock initialized: now 1781852037189005 us; error 54 us; skew 500 ppm
I20260619 06:53:57.190999 32215 webserver.cc:460] Webserver started at http://127.22.113.62:42149/ using document root <none> and password file <none>
I20260619 06:53:57.191674 32215 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:57.191782 32215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:57.192025 32215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:57.194142 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "762a49bbd47d4c58bc38d8745089308f"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "8afe237fb1bfef341a390f969fc212fa"
server_key_iv: "4b44ee6e4145f2cdf0d1fea91adf2f62"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.194747 32215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "762a49bbd47d4c58bc38d8745089308f"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "8afe237fb1bfef341a390f969fc212fa"
server_key_iv: "4b44ee6e4145f2cdf0d1fea91adf2f62"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.198750 32215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:53:57.201553 32229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.203007 32215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:57.203172 32215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "762a49bbd47d4c58bc38d8745089308f"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "8afe237fb1bfef341a390f969fc212fa"
server_key_iv: "4b44ee6e4145f2cdf0d1fea91adf2f62"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.203354 32215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:57.221736 32215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:57.222494 32215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:57.222693 32215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:57.230145 32215 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:46059
I20260619 06:53:57.230175 32281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:46059 every 8 connection(s)
I20260619 06:53:57.231145 32215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:57.232498 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32215
I20260619 06:53:57.232637 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:57.232931 22980 external_mini_cluster.cc:1428] Setting key a0d409559b95c51e301325bcb5e838d0
I20260619 06:53:57.235430 32282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:57.242787 32282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: Bootstrap starting.
I20260619 06:53:57.243067 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.234509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.22.113.62:46059)
0619 06:53:57.235377 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:57.235395 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:57.235434 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:53:57.236203 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:57.236222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:57.236673 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:57.237031 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:57.237459 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.238562 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.239499 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.239990 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.240172 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.241204 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:57.241209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:57.241221 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:57.241625 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:57.241682 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:57.241702 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:57.241818 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:57.241976 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:53:57.242238 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":192,"threads_started":1}
I20260619 06:53:57.245620 32282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:57.246484 32282 log.cc:824] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:57.248849 32282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: No bootstrap required, opened a new log
I20260619 06:53:57.252965 32282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } }
I20260619 06:53:57.253208 32282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:57.253288 32282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762a49bbd47d4c58bc38d8745089308f, State: Initialized, Role: FOLLOWER
I20260619 06:53:57.253810 32282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } }
I20260619 06:53:57.253983 32282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:57.254055 32282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:57.254199 32282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:57.255223 32282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } }
I20260619 06:53:57.255669 32282 leader_election.cc:302] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 762a49bbd47d4c58bc38d8745089308f; no voters:
I20260619 06:53:57.256011 32282 leader_election.cc:288] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:57.256203 32287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:57.256575 32287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [term 1 LEADER]: Becoming Leader. State: Replica: 762a49bbd47d4c58bc38d8745089308f, State: Running, Role: LEADER
I20260619 06:53:57.256975 32287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } }
I20260619 06:53:57.257448 32282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:57.258951 32288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "762a49bbd47d4c58bc38d8745089308f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } } }
I20260619 06:53:57.258989 32289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 762a49bbd47d4c58bc38d8745089308f. Latest consensus state: current_term: 1 leader_uuid: "762a49bbd47d4c58bc38d8745089308f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "762a49bbd47d4c58bc38d8745089308f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46059 } } }
I20260619 06:53:57.259116 32289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:57.259116 32288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f [sys.catalog]: This master's current role is: LEADER
W20260619 06:53:57.263096 32302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:53:57.263190 32302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:53:57.263290 32297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:57.264006 32297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:57.270255 32297 catalog_manager.cc:1347] Generated new cluster ID: 26343d45e92a4107b79df85008a801b6
I20260619 06:53:57.270323 32297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:57.277500 32297 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:57.278810 32297 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:57.288079 32297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 762a49bbd47d4c58bc38d8745089308f: Generated new TSK 0
I20260619 06:53:57.288897 32297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:57.295312 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--builtin_ntp_servers=127.22.113.20:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:53:57.427768 32306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:57.428052 32306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:57.428093 32306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:57.432260 32306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:57.432313 32306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:57.432366 32306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:57.437320 32306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:57.438572 32306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:57.440214 32306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:57.451299 32312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:57.451354 32311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.451500 32306 server_base.cc:1034] running on GCE node
W20260619 06:53:57.451676 32314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.452203 32306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:57.452948 32306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:57.454146 32306 hybrid_clock.cc:648] HybridClock initialized: now 1781852037454108 us; error 54 us; skew 500 ppm
I20260619 06:53:57.456221 32306 webserver.cc:460] Webserver started at http://127.22.113.1:39303/ using document root <none> and password file <none>
I20260619 06:53:57.456856 32306 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:57.456950 32306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:57.457232 32306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:57.459653 32306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "e680412670ea4573ade2f8d4b6e53519"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "39ab7e293d1d4a731ff6f65b3a88639d"
server_key_iv: "ea1998c167db38b7efd52ebfc01ed831"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.460234 32306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "e680412670ea4573ade2f8d4b6e53519"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "39ab7e293d1d4a731ff6f65b3a88639d"
server_key_iv: "ea1998c167db38b7efd52ebfc01ed831"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.464242 32306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:57.466895 32320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.468262 32306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:57.468415 32306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "e680412670ea4573ade2f8d4b6e53519"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "39ab7e293d1d4a731ff6f65b3a88639d"
server_key_iv: "ea1998c167db38b7efd52ebfc01ed831"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.468538 32306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:57.477397 32306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:57.478040 32306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:57.478235 32306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:57.478807 32306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:57.479801 32306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:57.479885 32306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.479961 32306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:57.480006 32306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.490056 32306 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:45269
I20260619 06:53:57.490082 32433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:45269 every 8 connection(s)
I20260619 06:53:57.491001 32306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:57.494311 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32306
I20260619 06:53:57.494437 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:57.494802 22980 external_mini_cluster.cc:1428] Setting key 138154031737605935dcdc7110a249b7
I20260619 06:53:57.497170 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--builtin_ntp_servers=127.22.113.20:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:53:57.502310 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.493246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:49061 (local address 127.22.113.62:46059)
0619 06:53:57.493474 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:57.493480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:57.494712 (+ 1232us) server_negotiation.cc:407] Connection header received
0619 06:53:57.496578 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:57.496582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:57.496635 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:57.496802 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:57.497933 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.498812 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.500156 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.500390 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.500509 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.501804 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:57.501807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:57.501809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:57.501820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:57.501838 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:57.501845 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:57.501920 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:57.502083 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 06:53:57.502167 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 06:53:57.503191 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.493673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46059 (local address 127.22.113.1:49061)
0619 06:53:57.494489 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:57.494555 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:57.496243 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:57.496971 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:57.496978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:57.497078 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:57.497779 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.497792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.498986 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.498990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.500014 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.500027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.500556 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.500561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.500727 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.501492 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:57.501494 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:57.501535 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:57.501607 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:57.501934 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:57.501947 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:57.501956 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:53:57.502083 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:53:57.502399 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260619 06:53:57.504773 32434 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46059
I20260619 06:53:57.505028 32434 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:57.505549 32434 heartbeater.cc:511] Master 127.22.113.62:46059 requested a full tablet report, sending...
I20260619 06:53:57.507105 32246 ts_manager.cc:194] Registered new tserver with Master: e680412670ea4573ade2f8d4b6e53519 (127.22.113.1:45269)
I20260619 06:53:57.508352 32246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.1:49061
W20260619 06:53:57.622152 32437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:57.622524 32437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:57.622593 32437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:57.626814 32437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:57.626907 32437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:57.626999 32437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:57.631959 32437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:57.633255 32437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:57.634908 32437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:57.645800 32442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:57.645860 32443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.646138 32437 server_base.cc:1034] running on GCE node
W20260619 06:53:57.645800 32445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.646875 32437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:57.647725 32437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:57.648931 32437 hybrid_clock.cc:648] HybridClock initialized: now 1781852037648895 us; error 53 us; skew 500 ppm
I20260619 06:53:57.650970 32437 webserver.cc:460] Webserver started at http://127.22.113.2:45515/ using document root <none> and password file <none>
I20260619 06:53:57.651685 32437 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:57.651783 32437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:57.652004 32437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:57.654155 32437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "d197b7a69ebf44d793ebf3e64a55a55d"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "2f05344230c35f2ae94c643a9823d414"
server_key_iv: "9f051b2e9d0a79025e73192f8a7b24bd"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.654772 32437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "d197b7a69ebf44d793ebf3e64a55a55d"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "2f05344230c35f2ae94c643a9823d414"
server_key_iv: "9f051b2e9d0a79025e73192f8a7b24bd"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.658962 32437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:53:57.662206 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.664134 32437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:53:57.664309 32437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "d197b7a69ebf44d793ebf3e64a55a55d"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "2f05344230c35f2ae94c643a9823d414"
server_key_iv: "9f051b2e9d0a79025e73192f8a7b24bd"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.664446 32437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:57.686357 32437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:57.687124 32437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:57.687390 32437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:57.688012 32437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:57.689009 32437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:57.689083 32437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.689162 32437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:57.689206 32437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.699178 32437 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:40809
I20260619 06:53:57.699368 32564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:40809 every 8 connection(s)
I20260619 06:53:57.700183 32437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:57.704829 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32437
I20260619 06:53:57.704962 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:57.705243 22980 external_mini_cluster.cc:1428] Setting key 052f1e681ae97500c3664e10b209fe3e
I20260619 06:53:57.707510 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--builtin_ntp_servers=127.22.113.20:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:53:57.714885 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.702440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:60213 (local address 127.22.113.62:46059)
0619 06:53:57.702643 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:57.702648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:57.703860 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 06:53:57.705549 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:57.705553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:57.705618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:57.705727 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:57.706965 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.707846 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.709195 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.709579 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.709716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.714252 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:57.714256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:57.714258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:57.714274 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:57.714300 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:57.714309 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:57.714409 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:57.714595 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:53:57.714691 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:53:57.716065 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.702820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46059 (local address 127.22.113.2:60213)
0619 06:53:57.703677 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:57.703727 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:57.705277 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:57.705964 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:57.705972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:57.706063 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:57.706798 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.706822 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.708001 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.708004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.709062 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.709071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.709752 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.709755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.709918 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.713903 (+ 3985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:57.713905 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:57.713958 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:57.714040 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:57.714422 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:57.714433 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:57.714446 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:53:57.714587 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:53:57.714933 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260619 06:53:57.717483 32565 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46059
I20260619 06:53:57.717774 32565 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:57.718416 32565 heartbeater.cc:511] Master 127.22.113.62:46059 requested a full tablet report, sending...
I20260619 06:53:57.719671 32246 ts_manager.cc:194] Registered new tserver with Master: d197b7a69ebf44d793ebf3e64a55a55d (127.22.113.2:40809)
I20260619 06:53:57.720335 32246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.2:60213
W20260619 06:53:57.844465 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:57.844774 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:57.844849 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:57.849022 32568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:57.849102 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:57.849188 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:57.853993 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:57.855124 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:57.856554 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:57.866670 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:57.866632 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:57.866644 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:57.867187 32568 server_base.cc:1034] running on GCE node
I20260619 06:53:57.867681 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:57.868332 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:57.869515 32568 hybrid_clock.cc:648] HybridClock initialized: now 1781852037869476 us; error 60 us; skew 500 ppm
I20260619 06:53:57.871470 32568 webserver.cc:460] Webserver started at http://127.22.113.3:43831/ using document root <none> and password file <none>
I20260619 06:53:57.872100 32568 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:57.872187 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:57.872422 32568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:57.874521 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "33c4c2305ad44b2597ae12579f8af697"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "a30f470a4e3705b11376a2e58169feb1"
server_key_iv: "9d1139028ec69e3197bd7256914663b1"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.875087 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "33c4c2305ad44b2597ae12579f8af697"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "a30f470a4e3705b11376a2e58169feb1"
server_key_iv: "9d1139028ec69e3197bd7256914663b1"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.878767 32568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:57.881496 32582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.882936 32568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:53:57.883083 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "33c4c2305ad44b2597ae12579f8af697"
format_stamp: "Formatted at 2026-06-19 06:53:57 on dist-test-slave-g8zg"
server_key: "a30f470a4e3705b11376a2e58169feb1"
server_key_iv: "9d1139028ec69e3197bd7256914663b1"
server_key_version: "encryptionkey@0"
I20260619 06:53:57.883203 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:57.906955 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:57.907768 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:57.907966 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:57.908557 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:57.909503 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:57.909582 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.909665 32568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:57.909716 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:57.919974 32568 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38209
I20260619 06:53:57.920046 32695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38209 every 8 connection(s)
I20260619 06:53:57.920941 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:57.924893 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32568
I20260619 06:53:57.925004 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:57.925345 22980 external_mini_cluster.cc:1428] Setting key 89256d20641d2f9b395c88cfab43d49b
I20260619 06:53:57.931563 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.923121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:41029 (local address 127.22.113.62:46059)
0619 06:53:57.923314 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:57.923319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:57.924598 (+ 1279us) server_negotiation.cc:407] Connection header received
0619 06:53:57.926313 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:57.926316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:57.926371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:57.926486 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:57.927692 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.928278 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.929610 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.929835 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.929920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.931118 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:57.931120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:57.931123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:57.931134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:57.931152 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:57.931159 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:57.931219 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:57.931391 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:53:57.931441 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:53:57.932590 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:57.923640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46059 (local address 127.22.113.3:41029)
0619 06:53:57.924427 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:57.924473 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:57.926082 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:57.926720 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:57.926732 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:57.926864 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:57.927514 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.927533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.928414 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.928419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.929486 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:53:57.929497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:53:57.929975 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:53:57.929978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:53:57.930128 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:53:57.930856 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:57.930857 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:57.930904 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:57.930981 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:57.931239 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:57.931263 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:57.931268 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:53:57.931375 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:53:57.931654 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260619 06:53:57.933840 32696 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46059
I20260619 06:53:57.934077 32696 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:57.934646 32696 heartbeater.cc:511] Master 127.22.113.62:46059 requested a full tablet report, sending...
I20260619 06:53:57.935854 32246 ts_manager.cc:194] Registered new tserver with Master: 33c4c2305ad44b2597ae12579f8af697 (127.22.113.3:38209)
I20260619 06:53:57.936518 32246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.3:41029
I20260619 06:53:57.939857 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:53:57.942890 32285 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:57.941890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.22.113.62:46059)
0619 06:53:57.942042 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:57.942049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:57.942244 (+ 195us) server_negotiation.cc:407] Connection header received
0619 06:53:57.942361 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:57.942363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:57.942406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:57.942478 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:57.942766 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35634: BlockingRecv error: recv got EOF from 127.0.0.1:35634 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:53:57.943189 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32306
I20260619 06:53:57.949307 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32437
I20260619 06:53:57.954660 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32568
I20260619 06:53:57.960623 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32215
2026-06-19T06:53:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (949 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:53:57Z Disabled control of system clock
I20260619 06:53:57.996462 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32793
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:32793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:53:58.125257 32707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:58.125568 32707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:58.125607 32707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:58.129851 32707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:53:58.129931 32707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:58.129947 32707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:58.129966 32707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:53:58.129982 32707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:53:58.135356 32707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:32793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:58.136685 32707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:58.138295 32707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:58.147794 32713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.147782 32712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.147795 32715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:58.148126 32707 server_base.cc:1034] running on GCE node
I20260619 06:53:58.148754 32707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:58.150018 32707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:58.151191 32707 hybrid_clock.cc:648] HybridClock initialized: now 1781852038151153 us; error 55 us; skew 500 ppm
I20260619 06:53:58.153368 32707 webserver.cc:460] Webserver started at http://127.22.113.62:42131/ using document root <none> and password file <none>
I20260619 06:53:58.154042 32707 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:58.154116 32707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:58.154359 32707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:58.156787 32707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "69a603a3ae2f4850b65a7ccc937db1bd"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "8502c85d6f734e61afca32eebb16ce16"
server_key_iv: "cbf2c59c42f31e6fa575aed0992a5315"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.157469 32707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "69a603a3ae2f4850b65a7ccc937db1bd"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "8502c85d6f734e61afca32eebb16ce16"
server_key_iv: "cbf2c59c42f31e6fa575aed0992a5315"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.161888 32707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:53:58.164848 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.166368 32707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:53:58.166522 32707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "69a603a3ae2f4850b65a7ccc937db1bd"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "8502c85d6f734e61afca32eebb16ce16"
server_key_iv: "cbf2c59c42f31e6fa575aed0992a5315"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.166662 32707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:58.171814 32707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:58.172478 32707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:58.172675 32707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:58.181041 32707 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:32793
I20260619 06:53:58.181123 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:32793 every 8 connection(s)
I20260619 06:53:58.182133 32707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:53:58.183792 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 32707
I20260619 06:53:58.183931 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:53:58.184324 22980 external_mini_cluster.cc:1428] Setting key af28e2774559644b85e018c4913ce43c
I20260619 06:53:58.186370 306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:53:58.190845 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.185904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.22.113.62:32793)
0619 06:53:58.186656 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:58.186675 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:58.186719 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:53:58.187620 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:58.187642 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:58.188065 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:58.188451 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:58.189333 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:58.189340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:58.189355 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:58.189367 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:58.189434 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:58.189459 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:58.189626 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:58.189698 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:53:58.189942 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260619 06:53:58.193354 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd: Bootstrap starting.
I20260619 06:53:58.195755 306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd: Neither blocks nor log segments found. Creating new log.
I20260619 06:53:58.196501 306 log.cc:824] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:53:58.198815 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd: No bootstrap required, opened a new log
I20260619 06:53:58.202534 306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } }
I20260619 06:53:58.202752 306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:53:58.202841 306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a603a3ae2f4850b65a7ccc937db1bd, State: Initialized, Role: FOLLOWER
I20260619 06:53:58.203315 306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } }
I20260619 06:53:58.203513 306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:53:58.203581 306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:53:58.203706 306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:53:58.204627 306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } }
I20260619 06:53:58.204978 306 leader_election.cc:302] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69a603a3ae2f4850b65a7ccc937db1bd; no voters:
I20260619 06:53:58.205289 306 leader_election.cc:288] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:53:58.205452 311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:53:58.205827 311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [term 1 LEADER]: Becoming Leader. State: Replica: 69a603a3ae2f4850b65a7ccc937db1bd, State: Running, Role: LEADER
I20260619 06:53:58.206208 311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } }
I20260619 06:53:58.206571 306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:53:58.208199 312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } } }
I20260619 06:53:58.208348 312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:58.208732 320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:53:58.208683 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a603a3ae2f4850b65a7ccc937db1bd. Latest consensus state: current_term: 1 leader_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a603a3ae2f4850b65a7ccc937db1bd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 32793 } } }
I20260619 06:53:58.208886 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd [sys.catalog]: This master's current role is: LEADER
I20260619 06:53:58.212095 320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:53:58.218207 320 catalog_manager.cc:1347] Generated new cluster ID: 7be59721007f4ecda6258bff71c4748e
I20260619 06:53:58.218290 320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:53:58.228318 320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:53:58.228688 320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:53:58.236203 320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69a603a3ae2f4850b65a7ccc937db1bd: Generated new TSK 0
I20260619 06:53:58.236966 320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:53:58.245059 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--builtin_ntp_servers=127.22.113.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:53:58.372607 330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:58.372973 330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:58.373020 330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:58.377388 330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:58.377452 330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:58.377516 330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:53:58.382586 330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:58.384009 330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:58.385825 330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:58.397163 336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.397184 335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.397197 338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:58.397547 330 server_base.cc:1034] running on GCE node
I20260619 06:53:58.398172 330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:58.398977 330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:58.400207 330 hybrid_clock.cc:648] HybridClock initialized: now 1781852038400167 us; error 70 us; skew 500 ppm
I20260619 06:53:58.402629 330 webserver.cc:460] Webserver started at http://127.22.113.1:36135/ using document root <none> and password file <none>
I20260619 06:53:58.403412 330 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:58.403512 330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:58.403828 330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:58.406299 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "7b2c741b3f2f43bfa01c9ad1d5308d51"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "ab61cd6d18bc0edcaa690a13ee4cf9b6"
server_key_iv: "1752ebb5c4081510110404ec54ffad97"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.406944 330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "7b2c741b3f2f43bfa01c9ad1d5308d51"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "ab61cd6d18bc0edcaa690a13ee4cf9b6"
server_key_iv: "1752ebb5c4081510110404ec54ffad97"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.412297 330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:53:58.415853 344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.417722 330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:58.417997 330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "7b2c741b3f2f43bfa01c9ad1d5308d51"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "ab61cd6d18bc0edcaa690a13ee4cf9b6"
server_key_iv: "1752ebb5c4081510110404ec54ffad97"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.418184 330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:58.423843 330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:58.424748 330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:58.424964 330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:58.425585 330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:58.426607 330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:58.426658 330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.426702 330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:58.426717 330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.438624 330 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:34415
I20260619 06:53:58.438661 457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:34415 every 8 connection(s)
I20260619 06:53:58.439924 330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:53:58.442821 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 330
I20260619 06:53:58.442957 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:53:58.443385 22980 external_mini_cluster.cc:1428] Setting key 814be747329624f680432039c466d39c
I20260619 06:53:58.446069 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--builtin_ntp_servers=127.22.113.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:53:58.448613 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.442470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:50085 (local address 127.22.113.62:32793)
0619 06:53:58.442732 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:58.442737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:58.444588 (+ 1851us) server_negotiation.cc:407] Connection header received
0619 06:53:58.446475 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:58.446479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:58.446525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:58.446630 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:58.448110 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:58.448114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:58.448117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:58.448132 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:58.448167 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:58.448177 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:58.448289 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:58.448428 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:53:58.448476 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 06:53:58.450604 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.443150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32793 (local address 127.22.113.1:50085)
0619 06:53:58.444411 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:58.444465 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:58.446128 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:58.446872 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:58.446884 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:58.447017 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:58.447820 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:58.447822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:58.447865 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:58.447945 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:58.448271 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:58.448279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:58.448285 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:53:58.448470 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 06:53:58.449767 (+ 1297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":152,"threads_started":1}
I20260619 06:53:58.452085 458 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32793
I20260619 06:53:58.452365 458 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:58.452818 458 heartbeater.cc:511] Master 127.22.113.62:32793 requested a full tablet report, sending...
I20260619 06:53:58.454921 32738 ts_manager.cc:194] Registered new tserver with Master: 7b2c741b3f2f43bfa01c9ad1d5308d51 (127.22.113.1:34415)
W20260619 06:53:58.574558 461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:58.574865 461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:58.574908 461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:58.579018 461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:58.579087 461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:58.579145 461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:53:58.584178 461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:58.585641 461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:58.587457 461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:58.598701 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:58.598831 461 server_base.cc:1034] running on GCE node
W20260619 06:53:58.598676 466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.599048 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:58.599647 461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:58.600425 461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:58.601610 461 hybrid_clock.cc:648] HybridClock initialized: now 1781852038601590 us; error 35 us; skew 500 ppm
I20260619 06:53:58.603839 461 webserver.cc:460] Webserver started at http://127.22.113.2:33987/ using document root <none> and password file <none>
I20260619 06:53:58.604499 461 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:58.604573 461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:58.604810 461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:58.607151 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "3a47057637df49bc9d8f81729b87de39"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "352a33b3e23c3b62e33ab8d55a81217f"
server_key_iv: "ea028dbb1ef045bb0313b8eee3a7e7b6"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.607884 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "3a47057637df49bc9d8f81729b87de39"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "352a33b3e23c3b62e33ab8d55a81217f"
server_key_iv: "ea028dbb1ef045bb0313b8eee3a7e7b6"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.612105 461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:53:58.614881 475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.616441 461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:58.616595 461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "3a47057637df49bc9d8f81729b87de39"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "352a33b3e23c3b62e33ab8d55a81217f"
server_key_iv: "ea028dbb1ef045bb0313b8eee3a7e7b6"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.616721 461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:58.621416 461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:58.622042 461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:58.622283 461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:58.622852 461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:58.623870 461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:58.623920 461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.623987 461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:58.624022 461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.633766 461 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:42327
I20260619 06:53:58.634044 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:42327 every 8 connection(s)
I20260619 06:53:58.634819 461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:53:58.641947 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.637188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:46855 (local address 127.22.113.62:32793)
0619 06:53:58.637327 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:58.637330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:58.638666 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 06:53:58.640154 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:58.640168 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:58.640216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:58.640322 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:58.641473 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:58.641477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:58.641479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:58.641497 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:58.641515 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:58.641522 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:58.641615 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:58.641766 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:53:58.641812 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:53:58.642829 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.637646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32793 (local address 127.22.113.2:46855)
0619 06:53:58.638477 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:58.638524 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:58.639931 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:58.640492 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:58.640500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:58.640592 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:58.641218 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:58.641220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:58.641259 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:58.641322 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:58.641607 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:58.641615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:58.641623 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:53:58.641815 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:53:58.642048 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":156,"threads_started":1}
I20260619 06:53:58.644141 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 461
I20260619 06:53:58.644276 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:53:58.644304 589 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32793
I20260619 06:53:58.644562 589 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:58.644611 22980 external_mini_cluster.cc:1428] Setting key 1f001999c8161148c91092ff70ab0b55
I20260619 06:53:58.645059 589 heartbeater.cc:511] Master 127.22.113.62:32793 requested a full tablet report, sending...
I20260619 06:53:58.646271 32737 ts_manager.cc:194] Registered new tserver with Master: 3a47057637df49bc9d8f81729b87de39 (127.22.113.2:42327)
I20260619 06:53:58.646972 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--builtin_ntp_servers=127.22.113.20:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:53:58.777329 592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:53:58.777657 592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:53:58.777702 592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:53:58.782646 592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:53:58.782739 592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:53:58.782805 592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:53:58.788280 592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:53:58.789753 592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:53:58.791493 592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:53:58.802976 600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.802994 597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:53:58.802994 598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:53:58.803575 592 server_base.cc:1034] running on GCE node
I20260619 06:53:58.804111 592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:53:58.804929 592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:53:58.806120 592 hybrid_clock.cc:648] HybridClock initialized: now 1781852038806098 us; error 36 us; skew 500 ppm
I20260619 06:53:58.808416 592 webserver.cc:460] Webserver started at http://127.22.113.3:35491/ using document root <none> and password file <none>
I20260619 06:53:58.809157 592 fs_manager.cc:359] Metadata directory not provided
I20260619 06:53:58.809226 592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:53:58.809481 592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:53:58.811929 592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "0b4bfd95eca94cd298cda08d8b8da20b"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "70221cfa662a37bec7c7dd577ed1cc17"
server_key_iv: "2812da9f663c251cd1ebbcdd118a7fea"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.812564 592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "0b4bfd95eca94cd298cda08d8b8da20b"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "70221cfa662a37bec7c7dd577ed1cc17"
server_key_iv: "2812da9f663c251cd1ebbcdd118a7fea"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.816906 592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:53:58.819962 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.821484 592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:53:58.821586 592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "0b4bfd95eca94cd298cda08d8b8da20b"
format_stamp: "Formatted at 2026-06-19 06:53:58 on dist-test-slave-g8zg"
server_key: "70221cfa662a37bec7c7dd577ed1cc17"
server_key_iv: "2812da9f663c251cd1ebbcdd118a7fea"
server_key_version: "encryptionkey@0"
I20260619 06:53:58.821741 592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:53:58.826867 592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:53:58.827634 592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:53:58.827829 592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:53:58.828430 592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:53:58.829455 592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:53:58.829509 592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.829581 592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:53:58.829623 592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:53:58.840643 592 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:42035
I20260619 06:53:58.840692 719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:42035 every 8 connection(s)
I20260619 06:53:58.841701 592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:53:58.844604 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 592
I20260619 06:53:58.844754 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:53:58.845109 22980 external_mini_cluster.cc:1428] Setting key 5a0836d04c001d94ededf77d54fbe63d
I20260619 06:53:58.849220 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.843982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46177 (local address 127.22.113.62:32793)
0619 06:53:58.844193 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:58.844198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:58.845561 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 06:53:58.847377 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:58.847380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:58.847439 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:58.847539 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:58.848778 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:53:58.848781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:53:58.848783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:53:58.848794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:53:58.848817 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:53:58.848825 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:53:58.848904 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:53:58.849053 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:53:58.849096 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 06:53:58.850160 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:53:58.844418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:32793 (local address 127.22.113.3:46177)
0619 06:53:58.845366 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 06:53:58.845435 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 06:53:58.847058 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:53:58.847769 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:53:58.847776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:53:58.847868 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:53:58.848504 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:53:58.848506 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:53:58.848549 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:53:58.848619 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:53:58.848917 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:53:58.848923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:53:58.848928 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:53:58.849075 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 06:53:58.849275 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":133,"threads_started":1}
I20260619 06:53:58.851438 720 heartbeater.cc:347] Connected to a master server at 127.22.113.62:32793
I20260619 06:53:58.851655 720 heartbeater.cc:464] Registering TS with master...
I20260619 06:53:58.852046 720 heartbeater.cc:511] Master 127.22.113.62:32793 requested a full tablet report, sending...
I20260619 06:53:58.853061 32737 ts_manager.cc:194] Registered new tserver with Master: 0b4bfd95eca94cd298cda08d8b8da20b (127.22.113.3:42035)
I20260619 06:53:58.859136 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:53:58.862228 309 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:53:58.861243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.22.113.62:32793)
0619 06:53:58.861358 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:53:58.861362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:53:58.861619 (+ 257us) server_negotiation.cc:407] Connection header received
0619 06:53:58.861721 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:53:58.861724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:53:58.861760 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:53:58.861828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:53:58.862078 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38478: BlockingRecv error: recv got EOF from 127.0.0.1:38478 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:53:58.862335 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 330
I20260619 06:53:58.868191 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 461
I20260619 06:53:58.874059 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 592
I20260619 06:53:58.880374 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 32707
2026-06-19T06:53:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (918 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:53:58 dist-test-slave-g8zg krb5kdc[730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:53:58 dist-test-slave-g8zg krb5kdc[730](info): set up 2 sockets
Jun 19 06:53:58 dist-test-slave-g8zg krb5kdc[730](info): commencing operation
krb5kdc: starting...
W20260619 06:54:00.033272 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:54:00.170222 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:46767
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:00.297696 743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:00.298069 743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:00.298162 743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:00.302670 743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:54:00.302802 743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:00.302829 743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:00.302848 743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:54:00.302865 743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:54:00.308869 743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:00.310169 743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:00.311398 743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:00.321516 748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:00.321604 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:00.321604 751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:00.321780 743 server_base.cc:1034] running on GCE node
I20260619 06:54:00.322682 743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:00.324055 743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:00.325251 743 hybrid_clock.cc:648] HybridClock initialized: now 1781852040325204 us; error 64 us; skew 500 ppm
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:00.328531 743 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:54:00.329594 743 webserver.cc:460] Webserver started at http://127.22.113.62:43149/ using document root <none> and password file <none>
I20260619 06:54:00.330317 743 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:00.330389 743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:00.330622 743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:54:00.332844 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "876db229c90b436394c10044077ecea8"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "1d095c60bdb28cc199b6767d42f9205e"
server_key_iv: "440291d30ab3921f70c8621382443d0b"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.333441 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "876db229c90b436394c10044077ecea8"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "1d095c60bdb28cc199b6767d42f9205e"
server_key_iv: "440291d30ab3921f70c8621382443d0b"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.337713 743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:54:00.340601 758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.342115 743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:54:00.342285 743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "876db229c90b436394c10044077ecea8"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "1d095c60bdb28cc199b6767d42f9205e"
server_key_iv: "440291d30ab3921f70c8621382443d0b"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.342412 743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:00.362135 743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:00.365504 743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:00.365723 743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:00.373890 743 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:46767
I20260619 06:54:00.373982 810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:46767 every 8 connection(s)
I20260619 06:54:00.374945 743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:54:00.377624 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 743
I20260619 06:54:00.377772 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:54:00.378038 22980 external_mini_cluster.cc:1428] Setting key 3723764a9798a6ebb39c5c5768d30a74
I20260619 06:54:00.378635 811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:00.386839 811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: Bootstrap starting.
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:00.390091 811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:00.391108 811 log.cc:824] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:00.393996 811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: No bootstrap required, opened a new log
I20260619 06:54:00.394881 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:00.379761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.22.113.62:46767)
0619 06:54:00.380261 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:00.380284 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:00.380337 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:54:00.381293 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:00.381319 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:00.381796 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:00.382236 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:00.382967 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.384543 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.385427 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.386159 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.386337 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:00.389266 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:00.389273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:00.389289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:00.389324 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:00.391499 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.392031 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.392036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.392041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.392187 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.392473 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.392477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.392479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.393021 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:00.393229 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:00.393491 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 06:54:00.393826 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":123,"threads_started":1}
I20260619 06:54:00.398566 811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } }
I20260619 06:54:00.398808 811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:00.398849 811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876db229c90b436394c10044077ecea8, State: Initialized, Role: FOLLOWER
I20260619 06:54:00.399367 811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } }
I20260619 06:54:00.399514 811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:54:00.399570 811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:54:00.399652 811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:00.400664 811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } }
I20260619 06:54:00.400995 811 leader_election.cc:302] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 876db229c90b436394c10044077ecea8; no voters:
I20260619 06:54:00.401305 811 leader_election.cc:288] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:54:00.401540 816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:54:00.401907 816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [term 1 LEADER]: Becoming Leader. State: Replica: 876db229c90b436394c10044077ecea8, State: Running, Role: LEADER
I20260619 06:54:00.402283 816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } }
I20260619 06:54:00.402884 811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:54:00.404399 817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "876db229c90b436394c10044077ecea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } } }
I20260619 06:54:00.404544 817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:54:00.404901 818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876db229c90b436394c10044077ecea8. Latest consensus state: current_term: 1 leader_uuid: "876db229c90b436394c10044077ecea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "876db229c90b436394c10044077ecea8" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 46767 } } }
I20260619 06:54:00.404991 818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:54:00.405099 825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:54:00.408361 825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:54:00.414060 825 catalog_manager.cc:1347] Generated new cluster ID: 4a26dba7a406478f8ded8ab3338ba1bd
I20260619 06:54:00.414129 825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:54:00.432911 825 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:54:00.433929 825 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:54:00.439762 825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: Generated new TSK 0
I20260619 06:54:00.440486 825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:54:00.470443 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:00.585690 837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:00.585955 837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:00.585999 837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:00.589944 837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:00.590001 837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:00.590057 837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:54:00.595278 837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:00.596349 837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:00.597318 837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:00.607707 842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:00.607707 843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:00.607913 837 server_base.cc:1034] running on GCE node
W20260619 06:54:00.607913 845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:00.608502 837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:00.609182 837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:00.610358 837 hybrid_clock.cc:648] HybridClock initialized: now 1781852040610336 us; error 40 us; skew 500 ppm
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:00.613333 837 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:54:00.614216 837 webserver.cc:460] Webserver started at http://127.22.113.1:35255/ using document root <none> and password file <none>
I20260619 06:54:00.614822 837 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:00.614888 837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:00.615125 837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:54:00.617141 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "94eb0e222aa74da0963dc99419d5e8dc"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "d5a47179cb896c20628d7270e0ba5b52"
server_key_iv: "c827ca661c40782293be35d50320cc2f"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.617704 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "94eb0e222aa74da0963dc99419d5e8dc"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "d5a47179cb896c20628d7270e0ba5b52"
server_key_iv: "c827ca661c40782293be35d50320cc2f"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.621439 837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:54:00.623941 852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.625269 837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:54:00.625413 837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "94eb0e222aa74da0963dc99419d5e8dc"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "d5a47179cb896c20628d7270e0ba5b52"
server_key_iv: "c827ca661c40782293be35d50320cc2f"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.625523 837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:00.658008 837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:00.661429 837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:00.661643 837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:00.662254 837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:00.663117 837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:00.663192 837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.663282 837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:00.663340 837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.673529 837 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:39755
I20260619 06:54:00.673558 965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:39755 every 8 connection(s)
I20260619 06:54:00.674458 837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:54:00.677212 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 837
I20260619 06:54:00.677350 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:54:00.677739 22980 external_mini_cluster.cc:1428] Setting key ff8e5b53e1a3460a48a7585aca907178
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:54:00.691179 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:00.677268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:46899 (local address 127.22.113.62:46767)
0619 06:54:00.677470 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:00.677475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:00.678666 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 06:54:00.680521 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:00.680525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:00.680579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:00.680659 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:00.681978 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.682551 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.683893 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.684150 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.684265 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:00.686985 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:00.686989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:00.686991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:00.687012 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:00.688544 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.689180 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.689185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.689187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.689267 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.689695 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.689699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.689701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.689924 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:00.690056 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:00.690744 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 06:54:00.690905 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:54:00.691974 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:00.677496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46767 (local address 127.22.113.1:46899)
0619 06:54:00.678474 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:00.678565 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:00.680286 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:00.680855 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:00.680862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:00.681269 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:00.681826 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:00.681839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.682677 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.682681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:00.683769 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:00.683778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.684316 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.684320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:00.684502 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:00.685183 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:00.685201 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:00.686777 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:00.688667 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:00.688673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:00.688688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:00.689078 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:00.689362 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:00.689366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:00.689368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:00.689589 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:00.690108 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:00.690114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:00.690456 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 06:54:00.690657 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 06:54:00.690974 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":127,"threads_started":1}
I20260619 06:54:00.693521 966 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46767
I20260619 06:54:00.693858 966 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:00.694480 966 heartbeater.cc:511] Master 127.22.113.62:46767 requested a full tablet report, sending...
I20260619 06:54:00.696377 775 ts_manager.cc:194] Registered new tserver with Master: 94eb0e222aa74da0963dc99419d5e8dc (127.22.113.1:39755)
I20260619 06:54:00.697687 775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:46899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:54:00.706494 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:00.815752 971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:00.816004 971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:00.816044 971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:00.819850 971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:00.819908 971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:00.819965 971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:54:00.825059 971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:00.825989 971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:00.826901 971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:00.836939 977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:00.836941 976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:00.836939 979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:00.837273 971 server_base.cc:1034] running on GCE node
I20260619 06:54:00.837838 971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:00.838632 971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:00.839943 971 hybrid_clock.cc:648] HybridClock initialized: now 1781852040839915 us; error 37 us; skew 500 ppm
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:00.843029 971 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:54:00.844101 971 webserver.cc:460] Webserver started at http://127.22.113.2:35481/ using document root <none> and password file <none>
I20260619 06:54:00.844743 971 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:00.844805 971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:00.845000 971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:54:00.847550 971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "2b796f79b18d46d087ed232ca8e86769"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "ee993e60fb3e67cc2b1108b1fc33dbee"
server_key_iv: "dc4867929d8e45e66672fe9777947c12"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.848110 971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "2b796f79b18d46d087ed232ca8e86769"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "ee993e60fb3e67cc2b1108b1fc33dbee"
server_key_iv: "dc4867929d8e45e66672fe9777947c12"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.852504 971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:54:00.855499 986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.857064 971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:54:00.857223 971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "2b796f79b18d46d087ed232ca8e86769"
format_stamp: "Formatted at 2026-06-19 06:54:00 on dist-test-slave-g8zg"
server_key: "ee993e60fb3e67cc2b1108b1fc33dbee"
server_key_iv: "dc4867929d8e45e66672fe9777947c12"
server_key_version: "encryptionkey@0"
I20260619 06:54:00.857350 971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:00.871774 971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:00.875200 971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:00.875468 971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:00.876204 971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:00.877207 971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:00.877286 971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.877362 971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:00.877424 971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:00.888738 971 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:46047
I20260619 06:54:00.888777 1099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:46047 every 8 connection(s)
I20260619 06:54:00.889839 971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:54:00.893561 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 971
I20260619 06:54:00.893745 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:54:00.894095 22980 external_mini_cluster.cc:1428] Setting key c4b3144ad1144de6013b229bd619f1c4
Jun 19 06:54:00 dist-test-slave-g8zg krb5kdc[730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852040, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:54:00.907351 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:00.892054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:35603 (local address 127.22.113.62:46767)
0619 06:54:00.892213 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:00.892217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:00.893244 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 06:54:00.895196 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:00.895199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:00.895310 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:00.895427 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:00.896914 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.897625 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.898996 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.899442 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.899569 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:00.902828 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:00.902833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:00.902838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:00.902868 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:00.905022 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.905637 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.905642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.905644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.905763 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:00.906113 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:00.906116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:00.906118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:00.906344 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:00.906457 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:00.906976 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 06:54:00.907129 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:54:00.908098 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:00.892422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46767 (local address 127.22.113.2:35603)
0619 06:54:00.893063 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:00.893117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:00.894969 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:00.895656 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:00.895666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:00.896147 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:00.896751 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:00.896765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.897827 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.897833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:00.898841 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:00.898854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:00.899645 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:00.899650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:00.899826 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:00.900617 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:00.900646 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:00.902628 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:00.905160 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:00.905166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:00.905178 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:00.905520 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:00.905870 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:00.905879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:00.905881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:00.906008 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:00.906460 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:00.906468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:00.906709 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 06:54:00.906930 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 06:54:00.907283 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:54:00.909432 1100 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46767
I20260619 06:54:00.909793 1100 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:00.910449 1100 heartbeater.cc:511] Master 127.22.113.62:46767 requested a full tablet report, sending...
I20260619 06:54:00.911782 775 ts_manager.cc:194] Registered new tserver with Master: 2b796f79b18d46d087ed232ca8e86769 (127.22.113.2:46047)
I20260619 06:54:00.912411 775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:35603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:54:00.926620 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:01.039971 1105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:01.040274 1105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:01.040347 1105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:01.044423 1105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:01.044518 1105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:01.044608 1105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:54:01.050019 1105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:46767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:01.051071 1105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:01.052130 1105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:01.062261 1110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:01.062286 1113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:01.062307 1111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:01.062505 1105 server_base.cc:1034] running on GCE node
I20260619 06:54:01.063161 1105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:01.064002 1105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:01.065182 1105 hybrid_clock.cc:648] HybridClock initialized: now 1781852041065159 us; error 40 us; skew 500 ppm
Jun 19 06:54:01 dist-test-slave-g8zg krb5kdc[730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852041, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:01.068536 1105 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:54:01.069589 1105 webserver.cc:460] Webserver started at http://127.22.113.3:45095/ using document root <none> and password file <none>
I20260619 06:54:01.070268 1105 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:01.070355 1105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:01.070570 1105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:54:01.073032 1105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "3d3e9fc90172416f81e9312a89ddc054"
format_stamp: "Formatted at 2026-06-19 06:54:01 on dist-test-slave-g8zg"
server_key: "8a28081826365aa15d3271ffc0f88dd6"
server_key_iv: "e94707d418dc3ce606d7c5c812e8571b"
server_key_version: "encryptionkey@0"
I20260619 06:54:01.073884 1105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "3d3e9fc90172416f81e9312a89ddc054"
format_stamp: "Formatted at 2026-06-19 06:54:01 on dist-test-slave-g8zg"
server_key: "8a28081826365aa15d3271ffc0f88dd6"
server_key_iv: "e94707d418dc3ce606d7c5c812e8571b"
server_key_version: "encryptionkey@0"
I20260619 06:54:01.078630 1105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:54:01.081885 1120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:01.083551 1105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:54:01.083722 1105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "3d3e9fc90172416f81e9312a89ddc054"
format_stamp: "Formatted at 2026-06-19 06:54:01 on dist-test-slave-g8zg"
server_key: "8a28081826365aa15d3271ffc0f88dd6"
server_key_iv: "e94707d418dc3ce606d7c5c812e8571b"
server_key_version: "encryptionkey@0"
I20260619 06:54:01.083853 1105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:01.112650 1105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:01.116722 1105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:01.116979 1105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:01.117750 1105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:01.118763 1105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:01.118818 1105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:01.118893 1105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:01.118933 1105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:01.130868 1105 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38563
I20260619 06:54:01.130947 1233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38563 every 8 connection(s)
I20260619 06:54:01.132155 1105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:54:01.134657 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 1105
I20260619 06:54:01.134820 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:54:01.135118 22980 external_mini_cluster.cc:1428] Setting key a00222320c1c708b77185bd5ead2a7fc
Jun 19 06:54:01 dist-test-slave-g8zg krb5kdc[730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852041, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:01.148782 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.134443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46083 (local address 127.22.113.62:46767)
0619 06:54:01.134701 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.134706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.136024 (+ 1318us) server_negotiation.cc:407] Connection header received
0619 06:54:01.137906 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.137910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.137968 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.138057 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:01.139430 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.140079 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.141513 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.141752 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.141849 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.144587 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:01.144590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:01.144593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:01.144618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:01.146299 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:01.147089 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:01.147092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:01.147093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:01.147176 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:01.147587 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:01.147589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:01.147591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:01.147793 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:01.147877 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.148515 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.148636 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:54:01.149864 1236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.135003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:46767 (local address 127.22.113.3:46083)
0619 06:54:01.135818 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.135905 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.137669 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.138242 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.138249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.138679 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:01.139206 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.139219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.140224 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.140227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.141373 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.141383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.141919 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.141922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.142063 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.142740 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:01.142766 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:01.144414 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:01.146446 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:01.146451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:01.146462 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:01.146946 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:01.147333 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:01.147336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:01.147337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:01.147478 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:01.147909 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:01.147916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:01.148218 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.148437 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.148860 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":195,"threads_started":1}
I20260619 06:54:01.151223 1234 heartbeater.cc:347] Connected to a master server at 127.22.113.62:46767
I20260619 06:54:01.151535 1234 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:01.152077 1234 heartbeater.cc:511] Master 127.22.113.62:46767 requested a full tablet report, sending...
I20260619 06:54:01.153184 775 ts_manager.cc:194] Registered new tserver with Master: 3d3e9fc90172416f81e9312a89ddc054 (127.22.113.3:38563)
I20260619 06:54:01.153838 775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:46083
I20260619 06:54:01.161466 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:54:01.172715 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.163741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.22.113.62:46767)
0619 06:54:01.163919 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.163924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.164128 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:54:01.164312 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.164316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.164374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.164517 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:01.165386 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.166003 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.166967 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.167224 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.167407 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.169090 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:01.169094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:01.169097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:01.169130 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:01.170771 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:01.171266 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:01.171272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:01.171275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:01.171362 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:01.171685 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:01.171688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:01.171690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:01.171953 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:01.172115 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.172366 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.172513 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:54:01.176229 775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:54:01.180143 775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:54:01.197078 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.190466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.22.113.3:38563)
0619 06:54:01.191483 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.191497 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.191519 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:54:01.191597 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.191602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.191810 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.191963 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:01.192522 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.193580 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.195473 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.196266 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.196413 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.196615 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.196749 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.196872 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":97,"threads_started":1}
I20260619 06:54:01.197072 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.190386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38563 (local address 127.0.0.1:33832)
0619 06:54:01.190955 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.191007 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.191148 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.192153 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.192161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.192183 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:01.192385 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.192396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.193713 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.193719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.195315 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.195326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.196408 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.196413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.196538 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.196579 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.196715 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.196872 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":17,"thread_start_us":108,"threads_started":1}
I20260619 06:54:01.197340 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.190752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:46047 (local address 127.0.0.1:42960)
0619 06:54:01.191497 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.191519 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.191633 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.192268 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.192271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.192283 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:01.192411 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.192417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.194296 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.194300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.195492 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.195498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.196886 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.196893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.196990 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.197019 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.197071 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.197153 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":43,"thread_start_us":131,"threads_started":1}
I20260619 06:54:01.198032 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.191169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.22.113.2:46047)
0619 06:54:01.191678 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.191694 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.191720 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:54:01.191814 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.191820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.192054 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.192282 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:01.192518 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.194092 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.195624 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.196576 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.197539 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.197698 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.197753 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.197882 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":162,"threads_started":1}
I20260619 06:54:01.198782 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.191213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39755 (local address 127.0.0.1:33908)
0619 06:54:01.191988 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.192004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.192109 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.192576 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.192578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.192595 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:01.192666 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.192671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.194489 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.194495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.195576 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.195587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.197587 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.197591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.197679 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.197699 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.198570 (+ 871us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.198660 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260619 06:54:01.198882 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.191648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33908 (local address 127.22.113.1:39755)
0619 06:54:01.191953 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.191977 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.192111 (+ 134us) server_negotiation.cc:407] Connection header received
0619 06:54:01.192234 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.192240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.192425 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.192607 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:01.192798 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.194363 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.195745 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.196590 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.196699 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.197646 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.198609 (+ 963us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.198722 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260619 06:54:01.201364 1168 tablet_service.cc:1467] Processing CreateTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 (DEFAULT_TABLE table=test-table [id=2246b0a4532d4972b77e4ab07e4a7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:01.201398 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 (DEFAULT_TABLE table=test-table [id=2246b0a4532d4972b77e4ab07e4a7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:01.201862 900 tablet_service.cc:1467] Processing CreateTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 (DEFAULT_TABLE table=test-table [id=2246b0a4532d4972b77e4ab07e4a7ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:01.202791 1168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:01.202831 1034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:01.203051 900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:01.209491 1251 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: Bootstrap starting.
I20260619 06:54:01.212142 1251 tablet_bootstrap.cc:654] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:01.212363 1253 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Bootstrap starting.
I20260619 06:54:01.213004 1251 log.cc:824] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:01.211756 1252 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: Bootstrap starting.
I20260619 06:54:01.214493 1253 tablet_bootstrap.cc:654] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:01.215442 1251 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: No bootstrap required, opened a new log
I20260619 06:54:01.215713 1251 ts_tablet_manager.cc:1397] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:54:01.215785 1253 log.cc:824] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:01.215900 1252 tablet_bootstrap.cc:654] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:01.216603 1252 log.cc:824] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:01.218461 1253 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: No bootstrap required, opened a new log
I20260619 06:54:01.218528 1252 tablet_bootstrap.cc:492] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: No bootstrap required, opened a new log
I20260619 06:54:01.218725 1252 ts_tablet_manager.cc:1397] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:54:01.218725 1253 ts_tablet_manager.cc:1397] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:54:01.220738 1251 raft_consensus.cc:348] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.221055 1251 raft_consensus.cc:374] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:01.221139 1251 raft_consensus.cc:729] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94eb0e222aa74da0963dc99419d5e8dc, State: Initialized, Role: FOLLOWER
I20260619 06:54:01.221716 1251 consensus_queue.cc:260] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.222584 966 heartbeater.cc:503] Master 127.22.113.62:46767 was elected leader, sending a full tablet report...
I20260619 06:54:01.222754 1252 raft_consensus.cc:348] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.222997 1251 ts_tablet_manager.cc:1428] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:54:01.223037 1252 raft_consensus.cc:374] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:01.223090 1252 raft_consensus.cc:729] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b796f79b18d46d087ed232ca8e86769, State: Initialized, Role: FOLLOWER
I20260619 06:54:01.223670 1252 consensus_queue.cc:260] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.223667 1253 raft_consensus.cc:348] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.224025 1253 raft_consensus.cc:374] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:01.224126 1253 raft_consensus.cc:729] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3e9fc90172416f81e9312a89ddc054, State: Initialized, Role: FOLLOWER
I20260619 06:54:01.224510 1100 heartbeater.cc:503] Master 127.22.113.62:46767 was elected leader, sending a full tablet report...
I20260619 06:54:01.224659 1253 consensus_queue.cc:260] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.225425 1234 heartbeater.cc:503] Master 127.22.113.62:46767 was elected leader, sending a full tablet report...
I20260619 06:54:01.225718 1253 ts_tablet_manager.cc:1428] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:54:01.228147 1252 ts_tablet_manager.cc:1428] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:54:01.258872 1258 raft_consensus.cc:482] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:54:01.259099 1258 raft_consensus.cc:504] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.260231 1258 leader_election.cc:288] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3e9fc90172416f81e9312a89ddc054 (127.22.113.3:38563), 94eb0e222aa74da0963dc99419d5e8dc (127.22.113.1:39755)
I20260619 06:54:01.265404 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.260661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38563 (local address 127.22.113.2:60087)
0619 06:54:01.260850 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.260871 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.260989 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.261325 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.261329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.261364 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:01.261540 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.261549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.262507 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.262514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.263955 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.263966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.264914 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.264918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.265026 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.265054 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.265124 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.265188 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":31}
I20260619 06:54:01.265451 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.260742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:60087 (local address 127.22.113.3:38563)
0619 06:54:01.260919 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.260927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.260946 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:54:01.261105 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.261111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.261166 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.261250 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:01.261676 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.262382 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.264137 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.264757 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.264909 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.264974 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.265191 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.265279 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:54:01.265744 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.261037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39755 (local address 127.22.113.2:54117)
0619 06:54:01.261672 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.261689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:01.261788 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:01.262031 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:01.262033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:01.262086 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:01.262265 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.262272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.263041 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.263045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.264300 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:01.264307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.265276 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.265279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:01.265373 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.265396 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:54:01.265494 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:54:01.265573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":3,"thread_start_us":194,"threads_started":1}
I20260619 06:54:01.265745 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.261252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:54117 (local address 127.22.113.1:39755)
0619 06:54:01.261392 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.261398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.261792 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:54:01.261860 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.261863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.261915 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.262002 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:01.262388 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.262926 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.264445 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.265136 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.265285 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.265334 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.265537 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.265631 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:54:01.266381 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4602ea9b70e74da2a0a3b77a2d7c6e18" candidate_uuid: "2b796f79b18d46d087ed232ca8e86769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3e9fc90172416f81e9312a89ddc054" is_pre_election: true
I20260619 06:54:01.266606 920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4602ea9b70e74da2a0a3b77a2d7c6e18" candidate_uuid: "2b796f79b18d46d087ed232ca8e86769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94eb0e222aa74da0963dc99419d5e8dc" is_pre_election: true
I20260619 06:54:01.266723 1188 raft_consensus.cc:2454] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b796f79b18d46d087ed232ca8e86769 in term 0.
I20260619 06:54:01.267021 920 raft_consensus.cc:2454] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b796f79b18d46d087ed232ca8e86769 in term 0.
I20260619 06:54:01.267349 989 leader_election.cc:302] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b796f79b18d46d087ed232ca8e86769, 3d3e9fc90172416f81e9312a89ddc054; no voters:
I20260619 06:54:01.267751 1258 raft_consensus.cc:2781] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:54:01.267891 1258 raft_consensus.cc:482] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:54:01.267966 1258 raft_consensus.cc:3037] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:01.269201 1258 raft_consensus.cc:504] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.269557 1258 leader_election.cc:288] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3e9fc90172416f81e9312a89ddc054 (127.22.113.3:38563), 94eb0e222aa74da0963dc99419d5e8dc (127.22.113.1:39755)
I20260619 06:54:01.270102 920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4602ea9b70e74da2a0a3b77a2d7c6e18" candidate_uuid: "2b796f79b18d46d087ed232ca8e86769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94eb0e222aa74da0963dc99419d5e8dc"
I20260619 06:54:01.270087 1188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4602ea9b70e74da2a0a3b77a2d7c6e18" candidate_uuid: "2b796f79b18d46d087ed232ca8e86769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3e9fc90172416f81e9312a89ddc054"
I20260619 06:54:01.270246 920 raft_consensus.cc:3037] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:01.270247 1188 raft_consensus.cc:3037] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:01.271893 920 raft_consensus.cc:2454] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b796f79b18d46d087ed232ca8e86769 in term 1.
I20260619 06:54:01.271888 1188 raft_consensus.cc:2454] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b796f79b18d46d087ed232ca8e86769 in term 1.
I20260619 06:54:01.272204 989 leader_election.cc:302] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b796f79b18d46d087ed232ca8e86769, 94eb0e222aa74da0963dc99419d5e8dc; no voters:
I20260619 06:54:01.272432 1258 raft_consensus.cc:2781] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:54:01.272701 1258 raft_consensus.cc:686] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 1 LEADER]: Becoming Leader. State: Replica: 2b796f79b18d46d087ed232ca8e86769, State: Running, Role: LEADER
I20260619 06:54:01.273105 1258 consensus_queue.cc:237] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } }
I20260619 06:54:01.277011 775 catalog_manager.cc:5495] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b796f79b18d46d087ed232ca8e86769 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "2b796f79b18d46d087ed232ca8e86769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b796f79b18d46d087ed232ca8e86769" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 46047 } health_report { overall_health: HEALTHY } } }
I20260619 06:54:01.296639 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.292785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.22.113.2:46047)
0619 06:54:01.292924 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:01.292928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:01.292942 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:54:01.293032 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:01.293034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:01.293101 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:01.293211 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:54:01.293520 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.294147 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.295025 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:01.295237 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:01.295411 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:01.295697 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:54:01.296228 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:54:01.296327 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:01.296423 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 06:54:01.296480 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:54:01.303761 920 raft_consensus.cc:1261] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 94eb0e222aa74da0963dc99419d5e8dc [term 1 FOLLOWER]: Refusing update from remote peer 2b796f79b18d46d087ed232ca8e86769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:54:01.303890 1188 raft_consensus.cc:1261] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 1 FOLLOWER]: Refusing update from remote peer 2b796f79b18d46d087ed232ca8e86769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:54:01.304603 1258 consensus_queue.cc:1035] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94eb0e222aa74da0963dc99419d5e8dc" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:54:01.304816 1261 consensus_queue.cc:1035] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3e9fc90172416f81e9312a89ddc054" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:54:01.312284 1264 mvcc.cc:205] Tried to move back new op lower bound from 7298465961171255296 to 7298465961057931264. Current Snapshot: MvccSnapshot[applied={T|T < 7298465961171255296}]
I20260619 06:54:01.323388 775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:54:01.323589 775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:54:01.326170 775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 876db229c90b436394c10044077ecea8: Sending DeleteTablet for 3 replicas of tablet 4602ea9b70e74da2a0a3b77a2d7c6e18
I20260619 06:54:01.327145 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:01 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42960
I20260619 06:54:01.327373 1168 tablet_service.cc:1514] Processing DeleteTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:01 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:33832
I20260619 06:54:01.327813 900 tablet_service.cc:1514] Processing DeleteTablet for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:01 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:33908
I20260619 06:54:01.328014 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 837
I20260619 06:54:01.331084 1274 tablet_replica.cc:330] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: stopping tablet replica
I20260619 06:54:01.331458 1274 raft_consensus.cc:2229] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:54:01.331795 1274 raft_consensus.cc:2258] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:54:01.332190 1273 tablet_replica.cc:330] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769: stopping tablet replica
I20260619 06:54:01.332441 1273 raft_consensus.cc:2229] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:54:01.332820 1273 raft_consensus.cc:2258] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 2b796f79b18d46d087ed232ca8e86769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:54:01.334266 1274 ts_tablet_manager.cc:1905] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:54:01.336663 1274 ts_tablet_manager.cc:1918] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:54:01.336808 1274 log.cc:1196] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/4602ea9b70e74da2a0a3b77a2d7c6e18
W20260619 06:54:01.336948 761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:39755 (error 108)
I20260619 06:54:01.337252 1274 ts_tablet_manager.cc:1939] T 4602ea9b70e74da2a0a3b77a2d7c6e18 P 3d3e9fc90172416f81e9312a89ddc054: Deleting consensus metadata
I20260619 06:54:01.338284 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.337942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:39755 (local address 127.0.0.1:33918)
0619 06:54:01.338098 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.338194 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:54:01.338318 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 971
W20260619 06:54:01.339294 761 catalog_manager.cc:4584] TS 94eb0e222aa74da0963dc99419d5e8dc (127.22.113.1:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4602ea9b70e74da2a0a3b77a2d7c6e18: Network error: Client connection negotiation failed: client connection to 127.22.113.1:39755: connect: Connection refused (error 111)
I20260619 06:54:01.339541 761 catalog_manager.cc:4841] TS 3d3e9fc90172416f81e9312a89ddc054 (127.22.113.3:38563): tablet 4602ea9b70e74da2a0a3b77a2d7c6e18 (table test-table [id=2246b0a4532d4972b77e4ab07e4a7ef6]) successfully deleted
I20260619 06:54:01.345367 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 1105
I20260619 06:54:01.346498 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:01.345594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:46047 (local address 127.0.0.1:42972)
0619 06:54:01.346261 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:01.346347 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548}
I20260619 06:54:01.351203 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 743
2026-06-19T06:54:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2473 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:54:01 dist-test-slave-g8zg krb5kdc[1278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:54:01 dist-test-slave-g8zg krb5kdc[1278](info): set up 2 sockets
Jun 19 06:54:01 dist-test-slave-g8zg krb5kdc[1278](info): commencing operation
krb5kdc: starting...
W20260619 06:54:02.428952 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:54:02 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:54:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangeradmin_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangeradmin_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangerlookup_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangerlookup_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
I20260619 06:54:02.618237 22980 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9tRik3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/postgres -l logfile start
2026-06-19 06:54:05.284 UTC [1308] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:54:05.284 UTC [1308] LOG: listening on IPv4 address "127.22.113.20", port 42115
2026-06-19 06:54:05.287 UTC [1308] LOG: listening on Unix socket "/tmp/.s.PGSQL.42115"
2026-06-19 06:54:05.313 UTC [1311] LOG: database system was shut down at 2026-06-19 06:54:03 UTC
2026-06-19 06:54:05.317 UTC [1308] LOG: database system is ready to accept connections
I20260619 06:54:06.284219 22980 mini_postgres.cc:96] Postgres bound to 42115
2026-06-19 06:54:06.291 UTC [1319] FATAL: database "slave" does not exist
127.22.113.20:42115 - accepting connections
I20260619 06:54:06.292676 22980 mini_ranger.cc:161] Starting Ranger...
I20260619 06:54:06.309942 22980 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:54:06.840730 22980 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:54:06.840842 22980 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin
2026-06-19 06:54:07,131 [I] DB FLAVOR :POSTGRES
2026-06-19 06:54:07,132 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:54:07,132 [I] Checking connection..
2026-06-19 06:54:07,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:54:07,405 [I] Checking connection passed.
2026-06-19 06:54:07,406 [I] --------- Verifying version history table ---------
2026-06-19 06:54:07,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:07,664 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:54:07,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:07,954 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:54:07,955 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:54:07,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:54:08.221 UTC [1383] WARNING: there is no transaction in progress
2026-06-19 06:54:08,228 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:54:08,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:08,488 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:54:08,488 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:54:08,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:54:08,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:54:09,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g8zg.c.gcp-upstream.internal','N') ;"
2026-06-19 06:54:09,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:54:09,556 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:54:09,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:54:09,810 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:54:09,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:54:10,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:54:10,318 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:54:10,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:54:11.344 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.362 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.379 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.396 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.416 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.435 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.495 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.503 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.521 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.533 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.545 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.562 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.573 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.581 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.588 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:11.595 UTC [1491] WARNING: there is no transaction in progress
2026-06-19 06:54:12,273 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:54:12,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:54:12,534 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:54:12,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:54:12,794 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:54:12,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:54:13,063 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:54:13,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:54:13,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:54:13,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g8zg.c.gcp-upstream.internal';"
2026-06-19 06:54:13,855 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:54:13,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:54:14,136 [I] DB_PATCHES have already been applied
I20260619 06:54:14.142997 22980 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9tRik3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:54:14.143081 22980 mini_ranger.cc:193] Using host: 127.22.113.20
I20260619 06:54:14.147475 22980 mini_ranger.cc:227] Ranger admin URL: http://127.22.113.20:38027
Jun 19, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:54:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:54:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:54:15.263018 22980 mini_ranger.cc:160] Time spent starting Ranger: real 8.970s user 0.000s sys 0.013s
Jun 19, 2026 6:54:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:54:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 19, 2026 6:54:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:54:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:54:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:54:38.215833 22980 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangerkms_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/rangerkms_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.113.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.113.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:54:38 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852078, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:54:38.314213 22980 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:54:38.326520 22980 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:54:39.273518 22980 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:54:39.273653 22980 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms
I20260619 06:54:39.273777 22980 mini_ranger_kms.cc:227] Using postgres at 127.22.113.20:42115
2026-06-19 06:54:39,479 [I] DB FLAVOR :POSTGRES
2026-06-19 06:54:39,479 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:54:39,480 [I] Checking connection
2026-06-19 06:54:39,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:54:39,768 [I] connection success
2026-06-19 06:54:39,768 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:54:39,768 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:54:39,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:54:40,027 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:54:40,027 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:54:40,027 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:54:40,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:42115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:54:40,295 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:54:40.804039 22980 mini_ranger_kms.cc:310] Created kms service
I20260619 06:54:41.057869 22980 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:54:41.107211 22980 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:54:41 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852078, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:54:41.541138 22980 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:54:41.541285 22980 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms:/tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:54:41.541309 22980 mini_ranger_kms.cc:240] Using host: 127.22.113.20
I20260619 06:54:41.546053 22980 mini_ranger_kms.cc:276] Ranger KMS PID: 1672
I20260619 06:54:41.546202 22980 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.113.20:36915
Jun 19, 2026 6:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 06:54:43 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852083, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:43.800046 22980 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.486s user 0.004s sys 0.187s
I20260619 06:54:43.800220 22980 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:54:43.800361 22980 mini_ranger_kms.cc:401] 127.22.113.20:36915/kms/v1/keys
Jun 19 06:54:44 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852083, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
Jun 19 06:54:44 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852083, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
Jun 19 06:54:46 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:54:46.966574 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:43841
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:43841
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:36915/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:47.081390 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:47.081683 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:47.081723 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:47.085848 1722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:54:47.085914 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:47.085940 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:47.085958 1722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:54:47.085973 1722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:54:47.091506 1722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.113.20:36915/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:43841
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:47.092532 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:47.093596 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:47.103078 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:47.103142 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:47.103127 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:47.103539 1722 server_base.cc:1034] running on GCE node
I20260619 06:54:47.104218 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:47.105516 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:47.106758 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781852087106731 us; error 44 us; skew 500 ppm
Jun 19 06:54:47 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852087, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:47.117569 1722 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:54:47.118742 1722 webserver.cc:460] Webserver started at http://127.22.113.62:45959/ using document root <none> and password file <none>
I20260619 06:54:47.119465 1722 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:47.119556 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:47.119805 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:54:47 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852087, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:54:47.181222 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "d85a016337064cc28d5e0068156702b0"
format_stamp: "Formatted at 2026-06-19 06:54:47 on dist-test-slave-g8zg"
server_key: "badf7eebf673937623806b9fae4e26dd"
server_key_iv: "cda35e9c6aadaf9e3d8ecca585a5ec3d"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:47.181871 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "d85a016337064cc28d5e0068156702b0"
format_stamp: "Formatted at 2026-06-19 06:54:47 on dist-test-slave-g8zg"
server_key: "badf7eebf673937623806b9fae4e26dd"
server_key_iv: "cda35e9c6aadaf9e3d8ecca585a5ec3d"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:47.186507 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:54:47.207018 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:47.208571 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:54:47.208696 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "d85a016337064cc28d5e0068156702b0"
format_stamp: "Formatted at 2026-06-19 06:54:47 on dist-test-slave-g8zg"
server_key: "badf7eebf673937623806b9fae4e26dd"
server_key_iv: "cda35e9c6aadaf9e3d8ecca585a5ec3d"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:47.208875 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:47.226243 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:47.229529 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:47.229749 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:47.238581 1722 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:43841
I20260619 06:54:47.238708 1789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:43841 every 8 connection(s)
I20260619 06:54:47.239758 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:54:47.242658 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:47.246184 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 1722
I20260619 06:54:47.246353 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:54:47.246742 22980 external_mini_cluster.cc:1428] Setting key 90f554c1dc59b95c09aa41b584640cf7
I20260619 06:54:47.248922 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: Bootstrap starting.
I20260619 06:54:47.251734 1790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:47.252629 1790 log.cc:824] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:47.255752 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: No bootstrap required, opened a new log
Jun 19 06:54:47 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:47.261252 1790 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } }
I20260619 06:54:47.261747 1790 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:47.261853 1790 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85a016337064cc28d5e0068156702b0, State: Initialized, Role: FOLLOWER
I20260619 06:54:47.262447 1790 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } }
I20260619 06:54:47.262635 1790 raft_consensus.cc:388] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:54:47.262707 1790 raft_consensus.cc:482] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:54:47.262809 1790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:47.262989 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:47.248855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.22.113.62:43841)
0619 06:54:47.249432 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:47.249452 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:47.249505 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:54:47.250380 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:47.250402 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:47.250824 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:47.251176 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:47.252312 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:47.253573 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:47.254496 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:47.254986 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:47.255176 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:47.257842 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:47.257847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:47.257862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:47.257891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:47.259687 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:47.260153 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:47.260159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:47.260164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:47.260265 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:47.260553 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:47.260557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:47.260559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:47.260996 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:47.261227 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:47.261470 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:54:47.262019 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":172,"threads_started":1}
I20260619 06:54:47.264225 1790 raft_consensus.cc:504] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } }
I20260619 06:54:47.264662 1790 leader_election.cc:302] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85a016337064cc28d5e0068156702b0; no voters:
I20260619 06:54:47.265046 1790 leader_election.cc:288] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:54:47.265585 1795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:54:47.265841 1795 raft_consensus.cc:686] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [term 1 LEADER]: Becoming Leader. State: Replica: d85a016337064cc28d5e0068156702b0, State: Running, Role: LEADER
I20260619 06:54:47.266223 1795 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } }
I20260619 06:54:47.266970 1790 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:54:47.268150 1797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85a016337064cc28d5e0068156702b0. Latest consensus state: current_term: 1 leader_uuid: "d85a016337064cc28d5e0068156702b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } } }
I20260619 06:54:47.268291 1797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:54:47.268618 1796 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d85a016337064cc28d5e0068156702b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a016337064cc28d5e0068156702b0" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 43841 } } }
I20260619 06:54:47.268692 1804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:54:47.268710 1796 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:54:47.272388 1804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:54:47.272938 1790 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:54:47.283434 1804 catalog_manager.cc:1347] Generated new cluster ID: 56f675ed74504b789e2cc830de3e26e9
I20260619 06:54:47.283615 1804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:54:47.296768 1804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:54:47.298358 1804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:54:47.311968 1804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: Generated new TSK 0
I20260619 06:54:47.312995 1804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:54:47.999527 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:36915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:48.150578 1835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:48.150983 1835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:48.151054 1835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:48.155833 1835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:48.155925 1835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:48.156040 1835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:54:48.162210 1835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.113.20:36915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:48.163784 1835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:48.165171 1835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:48.176713 1840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:48.176589 1841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:48.177294 1835 server_base.cc:1034] running on GCE node
W20260619 06:54:48.177472 1843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:48.177925 1835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:48.178745 1835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:48.179921 1835 hybrid_clock.cc:648] HybridClock initialized: now 1781852088179882 us; error 62 us; skew 500 ppm
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:48.183488 1835 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:54:48.184906 1835 webserver.cc:460] Webserver started at http://127.22.113.1:37333/ using document root <none> and password file <none>
I20260619 06:54:48.185773 1835 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:48.185889 1835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:48.186182 1835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:54:48.205502 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "7f9358d74fff4e278542b54df87b6946"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "d6282b15701f5c40a934a42e82f20a14"
server_key_iv: "c7c57363d210281f7bfc8208281a7a80"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.206272 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "7f9358d74fff4e278542b54df87b6946"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "d6282b15701f5c40a934a42e82f20a14"
server_key_iv: "c7c57363d210281f7bfc8208281a7a80"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.211207 1835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:54:48.231701 1852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.233441 1835 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260619 06:54:48.233659 1835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "7f9358d74fff4e278542b54df87b6946"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "d6282b15701f5c40a934a42e82f20a14"
server_key_iv: "c7c57363d210281f7bfc8208281a7a80"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.233884 1835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:48.266801 1835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:48.270859 1835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:48.271075 1835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:48.271844 1835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:48.272979 1835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:48.273051 1835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.273105 1835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:48.273138 1835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.285583 1835 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:42715
I20260619 06:54:48.286823 1835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:54:48.287756 1965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:42715 every 8 connection(s)
I20260619 06:54:48.288630 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 1835
I20260619 06:54:48.288793 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:54:48.289186 22980 external_mini_cluster.cc:1428] Setting key fc02013f5a35766a831e8e04a8d8203e
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:48.305630 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:48.290233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:55523 (local address 127.22.113.62:43841)
0619 06:54:48.290627 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:48.290637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:48.291196 (+ 559us) server_negotiation.cc:407] Connection header received
0619 06:54:48.292478 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:48.292482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:48.292544 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:48.292610 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:48.294298 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.295196 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.296983 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.297368 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.297451 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:48.300724 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:48.300728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:48.300731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:48.300763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:48.302726 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:48.303478 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:48.303483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:48.303486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:48.303597 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:48.304038 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:48.304042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:48.304046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:48.304321 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:48.304411 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:48.305168 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 06:54:48.305341 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260619 06:54:48.307569 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:48.290528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43841 (local address 127.22.113.1:55523)
0619 06:54:48.291033 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:48.291086 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:48.292202 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:48.293036 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:48.293045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:48.293424 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:48.294105 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:48.294125 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.295414 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.295418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:48.296834 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:48.296847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.297681 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.297685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:48.297890 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:48.298671 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:48.298693 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:48.300498 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:48.302882 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:48.302889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:48.302910 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:48.303345 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:48.303719 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:48.303723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:48.303728 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:48.303926 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:48.304484 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:48.304493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:48.304877 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 06:54:48.305908 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0619 06:54:48.306320 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":158,"threads_started":1}
I20260619 06:54:48.308997 1966 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43841
I20260619 06:54:48.309252 1966 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:48.309901 1966 heartbeater.cc:511] Master 127.22.113.62:43841 requested a full tablet report, sending...
I20260619 06:54:48.311839 1754 ts_manager.cc:194] Registered new tserver with Master: 7f9358d74fff4e278542b54df87b6946 (127.22.113.1:42715)
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:54:48.313853 1754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:55523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:54:48.346496 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:36915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:54:48.491351 1974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:48.491803 1974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:48.491879 1974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:48.498128 1974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:48.498256 1974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:48.498365 1974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:54:48.506861 1974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.113.20:36915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:48.508344 1974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:48.509768 1974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:48.523185 1980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:48.523337 1983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:48.523458 1981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:48.524854 1974 server_base.cc:1034] running on GCE node
I20260619 06:54:48.525422 1974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:48.526355 1974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:48.527746 1974 hybrid_clock.cc:648] HybridClock initialized: now 1781852088527704 us; error 54 us; skew 500 ppm
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:48.532027 1974 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:54:48.533610 1974 webserver.cc:460] Webserver started at http://127.22.113.2:34691/ using document root <none> and password file <none>
I20260619 06:54:48.534561 1974 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:48.534667 1974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:48.534981 1974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:54:48.558106 1974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "9b5e99e435274797826dfb5849f847d9"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "61f0176b25243fb3831407e2e9b2b62d"
server_key_iv: "72f215fbbeccf50849007818ca6c6096"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.558971 1974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "9b5e99e435274797826dfb5849f847d9"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "61f0176b25243fb3831407e2e9b2b62d"
server_key_iv: "72f215fbbeccf50849007818ca6c6096"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.569042 1974 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.004s
I20260619 06:54:48.594071 1990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.595786 1974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:54:48.595942 1974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "9b5e99e435274797826dfb5849f847d9"
format_stamp: "Formatted at 2026-06-19 06:54:48 on dist-test-slave-g8zg"
server_key: "61f0176b25243fb3831407e2e9b2b62d"
server_key_iv: "72f215fbbeccf50849007818ca6c6096"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:48.596140 1974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:48.623914 1974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:48.627990 1974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:48.628221 1974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:48.628974 1974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:48.630112 1974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:48.630234 1974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.630321 1974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:48.630365 1974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:48.647976 1974 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:45139
I20260619 06:54:48.649276 1974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:54:48.655948 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 1974
I20260619 06:54:48.656117 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:54:48.656488 22980 external_mini_cluster.cc:1428] Setting key 4bda3d410f0e1599a93e2dc8c3989c07
I20260619 06:54:48.651408 2103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:45139 every 8 connection(s)
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:48.671178 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:48.655104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:50561 (local address 127.22.113.62:43841)
0619 06:54:48.655268 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:48.655274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:48.656086 (+ 812us) server_negotiation.cc:407] Connection header received
0619 06:54:48.657440 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:48.657444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:48.657512 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:48.657576 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:48.659573 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.660446 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.662157 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.662513 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.662602 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:48.666034 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:48.666038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:48.666041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:48.666073 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:48.668047 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:48.668920 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:48.668924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:48.668926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:48.669023 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:48.669616 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:48.669620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:48.669622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:48.669927 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:48.670029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:48.670817 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 06:54:48.670977 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:54:48.672989 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:48.653895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43841 (local address 127.22.113.2:50561)
0619 06:54:48.655897 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:48.655959 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:48.657094 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:48.658108 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:48.658120 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:48.658527 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:48.659369 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:48.659396 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.660627 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.660630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:48.661998 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:48.662011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:48.662838 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:48.662843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:48.663055 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:48.663942 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:48.663966 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:48.665798 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:48.668316 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:48.668324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:48.668343 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:48.668764 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:48.669149 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:48.669152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:48.669155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:48.669485 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:48.670109 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:48.670116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:48.670492 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 06:54:48.671495 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0619 06:54:48.671873 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"thread_start_us":172,"threads_started":1}
I20260619 06:54:48.674460 2104 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43841
I20260619 06:54:48.674731 2104 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:48.675400 2104 heartbeater.cc:511] Master 127.22.113.62:43841 requested a full tablet report, sending...
I20260619 06:54:48.676707 1754 ts_manager.cc:194] Registered new tserver with Master: 9b5e99e435274797826dfb5849f847d9 (127.22.113.2:45139)
I20260619 06:54:48.677551 1754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:50561
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:54:48.731412 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:36915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:54:48.910770 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:54:48.911654 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:54:48.911794 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:54:48.917927 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:54:48.918111 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:54:48.918249 2109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:54:48.926367 2109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.113.20:36915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:54:48.928272 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:54:48.929863 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:54:48.944082 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:54:48.944746 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:48.946049 2109 server_base.cc:1034] running on GCE node
W20260619 06:54:48.946303 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:54:48.946943 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:54:48.948494 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:54:48.950129 2109 hybrid_clock.cc:648] HybridClock initialized: now 1781852088950040 us; error 103 us; skew 500 ppm
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:54:48.956447 2109 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:54:48.958211 2109 webserver.cc:460] Webserver started at http://127.22.113.3:34345/ using document root <none> and password file <none>
I20260619 06:54:48.959108 2109 fs_manager.cc:359] Metadata directory not provided
I20260619 06:54:48.959237 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:54:48.959578 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
Jun 19 06:54:48 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:54:49.008378 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "1fcf3749f67c403a9eb8ace0d18e8422"
format_stamp: "Formatted at 2026-06-19 06:54:49 on dist-test-slave-g8zg"
server_key: "ded87cc15576429716fc2eb5d99a0049"
server_key_iv: "300777d8d9bcb19fd30649b436cee5f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:49.009295 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "1fcf3749f67c403a9eb8ace0d18e8422"
format_stamp: "Formatted at 2026-06-19 06:54:49 on dist-test-slave-g8zg"
server_key: "ded87cc15576429716fc2eb5d99a0049"
server_key_iv: "300777d8d9bcb19fd30649b436cee5f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:49.017994 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260619 06:54:49.049182 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:49.056159 2109 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.001s
I20260619 06:54:49.056391 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "1fcf3749f67c403a9eb8ace0d18e8422"
format_stamp: "Formatted at 2026-06-19 06:54:49 on dist-test-slave-g8zg"
server_key: "ded87cc15576429716fc2eb5d99a0049"
server_key_iv: "300777d8d9bcb19fd30649b436cee5f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:54:49.056587 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:54:49.122706 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:54:49.126631 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:54:49.126847 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:54:49.127689 2109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:54:49.128785 2109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:54:49.128852 2109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:49.128919 2109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:54:49.128954 2109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:54:49.142970 2109 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:35989
I20260619 06:54:49.144887 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:54:49.145664 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 2109
I20260619 06:54:49.145834 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:54:49.145823 2237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:35989 every 8 connection(s)
I20260619 06:54:49.146210 22980 external_mini_cluster.cc:1428] Setting key f4f256eb7f5c68bd3cd6049ff3b02a63
Jun 19 06:54:49 dist-test-slave-g8zg krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852088, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:54:49.164201 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.149470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:57111 (local address 127.22.113.62:43841)
0619 06:54:49.149642 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.149648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.150617 (+ 969us) server_negotiation.cc:407] Connection header received
0619 06:54:49.151920 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.151925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.151984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.152040 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:49.153672 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.154521 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.156254 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.156599 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.156675 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.159910 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:49.159914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:49.159917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:49.159949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:49.161652 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:49.162301 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:49.162305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:49.162307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:49.162407 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:49.162900 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:49.162904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:49.162906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:49.163134 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:49.163212 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.163908 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.164041 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:54:49.165843 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.148366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:43841 (local address 127.22.113.3:57111)
0619 06:54:49.150440 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.150497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.151652 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.152450 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.152460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.152837 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:54:49.153501 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.153519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.154678 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.154681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.156113 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.156124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.156903 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.156906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.157096 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.157888 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:54:49.157910 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:54:49.159678 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:54:49.161819 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:49.161825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:49.161843 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:49.162187 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:49.162608 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:54:49.162610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:54:49.162612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:54:49.162798 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:54:49.163298 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:54:49.163305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:54:49.163621 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.164441 (+ 820us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.164799 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"thread_start_us":125,"threads_started":1}
I20260619 06:54:49.167173 2238 heartbeater.cc:347] Connected to a master server at 127.22.113.62:43841
I20260619 06:54:49.167444 2238 heartbeater.cc:464] Registering TS with master...
I20260619 06:54:49.168042 2238 heartbeater.cc:511] Master 127.22.113.62:43841 requested a full tablet report, sending...
I20260619 06:54:49.169211 1754 ts_manager.cc:194] Registered new tserver with Master: 1fcf3749f67c403a9eb8ace0d18e8422 (127.22.113.3:35989)
I20260619 06:54:49.169947 1754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:57111
I20260619 06:54:49.173555 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:54:49.188894 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.177037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.22.113.62:43841)
0619 06:54:49.177192 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.177198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.177905 (+ 707us) server_negotiation.cc:407] Connection header received
0619 06:54:49.178133 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.178137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.178196 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.178252 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:54:49.179486 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.180472 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.181861 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.182227 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.182313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.184558 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:54:49.184562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:54:49.184566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:54:49.184596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:54:49.186636 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:49.187228 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:49.187233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:49.187235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:49.187354 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:54:49.187722 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:54:49.187725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:54:49.187727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:54:49.187980 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:54:49.188116 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.188537 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.188688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:54:49.199121 1754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:54:49.210951 1754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:54:49.237982 2255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.227824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:35989 (local address 127.0.0.1:35814)
0619 06:54:49.230496 (+ 2672us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.230510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.230597 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.232131 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.232134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.232150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:49.232288 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.232295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.233729 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.233732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.235894 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.235906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.237351 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.237355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.237491 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.237536 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.237655 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.237798 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2612,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260619 06:54:49.239631 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.228017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.22.113.3:35989)
0619 06:54:49.230352 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.230364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.230697 (+ 333us) server_negotiation.cc:407] Connection header received
0619 06:54:49.230776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.230781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.231841 (+ 1060us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.231992 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:49.232418 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.233595 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.236061 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.236998 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.237105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.237271 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.239307 (+ 2036us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.239425 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2214,"thread_start_us":112,"threads_started":1}
I20260619 06:54:49.242146 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.226799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:42715 (local address 127.0.0.1:35836)
0619 06:54:49.228698 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.228766 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.228946 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.229901 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.229909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.229936 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:49.230175 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.230184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.235717 (+ 5533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.235721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.237121 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.237134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.241594 (+ 4460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.241598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.241718 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.241745 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.241825 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.241928 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"mutex_wait_us":4,"thread_start_us":164,"threads_started":1}
I20260619 06:54:49.243116 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.227547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.22.113.2:45139)
0619 06:54:49.228373 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.228380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.229384 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 06:54:49.229483 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.229489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.229733 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.229902 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:49.230979 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.232338 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.238744 (+ 6406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.240158 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.240265 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.240468 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.242770 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.242924 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":77,"threads_started":1}
I20260619 06:54:49.243825 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.227133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45139 (local address 127.0.0.1:43866)
0619 06:54:49.229122 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.229139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.229252 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.230071 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.230075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.230093 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:49.230813 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.230821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.232514 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.232518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.238574 (+ 6056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.238586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.240596 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.240603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.240727 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.240757 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.243564 (+ 2807us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.243665 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1900,"mutex_wait_us":4875,"thread_start_us":640,"threads_started":1}
I20260619 06:54:49.248490 2031 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e06b08875402eaadd3495b106a633 (DEFAULT_TABLE table=test-table [id=420fd2e5c41349819c3b8aad48cd82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:49.251447 2031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e06b08875402eaadd3495b106a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:49.253947 2172 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e06b08875402eaadd3495b106a633 (DEFAULT_TABLE table=test-table [id=420fd2e5c41349819c3b8aad48cd82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:49.255283 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e06b08875402eaadd3495b106a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:49.257983 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.226984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.22.113.1:42715)
0619 06:54:49.228166 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.228175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.229141 (+ 966us) server_negotiation.cc:407] Connection header received
0619 06:54:49.229243 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.229249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.229519 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.229696 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:49.230326 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.235565 (+ 5239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.237290 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.241250 (+ 3960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.241375 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.241523 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.257544 (+ 16021us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.257764 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":122,"threads_started":1}
I20260619 06:54:49.263881 1894 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e06b08875402eaadd3495b106a633 (DEFAULT_TABLE table=test-table [id=420fd2e5c41349819c3b8aad48cd82a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:54:49.265399 1894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e06b08875402eaadd3495b106a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:54:49.284003 2257 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Bootstrap starting.
I20260619 06:54:49.287824 2259 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Bootstrap starting.
I20260619 06:54:49.291046 2259 tablet_bootstrap.cc:654] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:49.291738 2258 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: Bootstrap starting.
I20260619 06:54:49.294279 2258 tablet_bootstrap.cc:654] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:49.295156 2258 log.cc:824] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:49.296311 2259 log.cc:824] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:49.300289 2257 tablet_bootstrap.cc:654] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Neither blocks nor log segments found. Creating new log.
I20260619 06:54:49.301678 2257 log.cc:824] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:54:49.306964 2259 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: No bootstrap required, opened a new log
I20260619 06:54:49.307498 2257 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: No bootstrap required, opened a new log
I20260619 06:54:49.307760 2257 ts_tablet_manager.cc:1397] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260619 06:54:49.308768 2258 tablet_bootstrap.cc:492] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: No bootstrap required, opened a new log
I20260619 06:54:49.308990 2258 ts_tablet_manager.cc:1397] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20260619 06:54:49.312641 2259 ts_tablet_manager.cc:1397] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20260619 06:54:49.313205 2257 raft_consensus.cc:348] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.313614 2257 raft_consensus.cc:374] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:49.313779 2257 raft_consensus.cc:729] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5e99e435274797826dfb5849f847d9, State: Initialized, Role: FOLLOWER
I20260619 06:54:49.314467 2257 consensus_queue.cc:260] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.316732 2259 raft_consensus.cc:348] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.317062 2259 raft_consensus.cc:374] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:49.317122 2259 raft_consensus.cc:729] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf3749f67c403a9eb8ace0d18e8422, State: Initialized, Role: FOLLOWER
I20260619 06:54:49.317414 1966 heartbeater.cc:503] Master 127.22.113.62:43841 was elected leader, sending a full tablet report...
I20260619 06:54:49.317595 2257 ts_tablet_manager.cc:1428] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 06:54:49.317665 2259 consensus_queue.cc:260] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.318053 2104 heartbeater.cc:503] Master 127.22.113.62:43841 was elected leader, sending a full tablet report...
I20260619 06:54:49.318786 2259 ts_tablet_manager.cc:1428] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 06:54:49.319378 2258 raft_consensus.cc:348] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.319674 2258 raft_consensus.cc:374] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:54:49.319741 2258 raft_consensus.cc:729] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9358d74fff4e278542b54df87b6946, State: Initialized, Role: FOLLOWER
I20260619 06:54:49.320219 2258 consensus_queue.cc:260] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.321377 2258 ts_tablet_manager.cc:1428] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260619 06:54:49.323439 2238 heartbeater.cc:503] Master 127.22.113.62:43841 was elected leader, sending a full tablet report...
I20260619 06:54:49.335582 2265 raft_consensus.cc:482] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:54:49.335835 2265 raft_consensus.cc:504] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.337342 2265 leader_election.cc:288] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fcf3749f67c403a9eb8ace0d18e8422 (127.22.113.3:35989), 9b5e99e435274797826dfb5849f847d9 (127.22.113.2:45139)
I20260619 06:54:49.348767 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.338244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:48587 (local address 127.22.113.2:45139)
0619 06:54:49.338392 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.338397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.340238 (+ 1841us) server_negotiation.cc:407] Connection header received
0619 06:54:49.340520 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.340523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.340583 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.340635 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:49.341120 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.341947 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.347233 (+ 5286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.347939 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.348050 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.348114 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.348518 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.348612 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:54:49.349310 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.337789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45139 (local address 127.22.113.1:48587)
0619 06:54:49.340111 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.340128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.340353 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.340790 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.340793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.340817 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:49.340980 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.340988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.342113 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.342116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.347066 (+ 4950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.347077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.348241 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.348244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.348346 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.348374 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.349037 (+ 663us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.349128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2236,"thread_start_us":129,"threads_started":1}
I20260619 06:54:49.350272 2058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e06b08875402eaadd3495b106a633" candidate_uuid: "7f9358d74fff4e278542b54df87b6946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5e99e435274797826dfb5849f847d9" is_pre_election: true
I20260619 06:54:49.350749 2058 raft_consensus.cc:2454] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f9358d74fff4e278542b54df87b6946 in term 0.
I20260619 06:54:49.351524 1855 leader_election.cc:302] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f9358d74fff4e278542b54df87b6946, 9b5e99e435274797826dfb5849f847d9; no voters:
I20260619 06:54:49.351948 2265 raft_consensus.cc:2781] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:54:49.352057 2265 raft_consensus.cc:482] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:54:49.352120 2265 raft_consensus.cc:3037] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:49.353622 2265 raft_consensus.cc:504] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.354058 2265 leader_election.cc:288] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [CANDIDATE]: Term 1 election: Requested vote from peers 1fcf3749f67c403a9eb8ace0d18e8422 (127.22.113.3:35989), 9b5e99e435274797826dfb5849f847d9 (127.22.113.2:45139)
I20260619 06:54:49.354688 2058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e06b08875402eaadd3495b106a633" candidate_uuid: "7f9358d74fff4e278542b54df87b6946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5e99e435274797826dfb5849f847d9"
I20260619 06:54:49.354859 2058 raft_consensus.cc:3037] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:49.356459 2058 raft_consensus.cc:2454] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f9358d74fff4e278542b54df87b6946 in term 1.
I20260619 06:54:49.356901 1855 leader_election.cc:302] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f9358d74fff4e278542b54df87b6946, 9b5e99e435274797826dfb5849f847d9; no voters:
I20260619 06:54:49.357110 2265 raft_consensus.cc:2781] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:54:49.357355 2265 raft_consensus.cc:686] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [term 1 LEADER]: Becoming Leader. State: Replica: 7f9358d74fff4e278542b54df87b6946, State: Running, Role: LEADER
I20260619 06:54:49.357797 2265 consensus_queue.cc:237] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } }
I20260619 06:54:49.361552 1754 catalog_manager.cc:5495] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f9358d74fff4e278542b54df87b6946 (127.22.113.1). New cstate: current_term: 1 leader_uuid: "7f9358d74fff4e278542b54df87b6946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9358d74fff4e278542b54df87b6946" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20260619 06:54:49.366096 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.339010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:37781 (local address 127.22.113.3:35989)
0619 06:54:49.339142 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.339147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.343602 (+ 4455us) server_negotiation.cc:407] Connection header received
0619 06:54:49.343810 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.343814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.343872 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.343924 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:54:49.344343 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.345209 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.349888 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.350566 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.350673 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.350753 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.365821 (+ 15068us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.365951 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:54:49.366611 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.338600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:35989 (local address 127.22.113.1:37781)
0619 06:54:49.343498 (+ 4898us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.343515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:54:49.343681 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:54:49.344061 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:54:49.344064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:54:49.344080 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:54:49.344202 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.344209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.345397 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.345401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.346671 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:54:49.346686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.350863 (+ 4177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.350868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:54:49.350995 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.351020 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:54:49.366329 (+ 15309us) client_negotiation.cc:240] Negotiation successful
0619 06:54:49.366440 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4801,"thread_start_us":111,"threads_started":1}
I20260619 06:54:49.367293 2185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e06b08875402eaadd3495b106a633" candidate_uuid: "7f9358d74fff4e278542b54df87b6946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" is_pre_election: true
I20260619 06:54:49.367642 2185 raft_consensus.cc:2454] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f9358d74fff4e278542b54df87b6946 in term 0.
I20260619 06:54:49.368026 2183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e06b08875402eaadd3495b106a633" candidate_uuid: "7f9358d74fff4e278542b54df87b6946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf3749f67c403a9eb8ace0d18e8422"
I20260619 06:54:49.368147 2183 raft_consensus.cc:3037] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:54:49.369560 2183 raft_consensus.cc:2454] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f9358d74fff4e278542b54df87b6946 in term 1.
W20260619 06:54:49.400159 2239 tablet.cc:2369] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:54:49.406352 2105 tablet.cc:2369] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:54:49.407742 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.401735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.22.113.1:42715)
0619 06:54:49.401923 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:54:49.401929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:54:49.402231 (+ 302us) server_negotiation.cc:407] Connection header received
0619 06:54:49.402338 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:54:49.402341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:54:49.402424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:54:49.402540 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:54:49.402967 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.403900 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.405297 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:54:49.405678 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:54:49.405803 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:54:49.406266 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:54:49.406924 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:54:49.407066 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:54:49.407470 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:54:49.407554 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:54:49.416415 2058 raft_consensus.cc:1261] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 1 FOLLOWER]: Refusing update from remote peer 7f9358d74fff4e278542b54df87b6946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:54:49.416687 2183 raft_consensus.cc:1261] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 1 FOLLOWER]: Refusing update from remote peer 7f9358d74fff4e278542b54df87b6946: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:54:49.417371 2268 consensus_queue.cc:1035] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5e99e435274797826dfb5849f847d9" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:54:49.417659 2265 consensus_queue.cc:1035] T ba2e06b08875402eaadd3495b106a633 P 7f9358d74fff4e278542b54df87b6946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf3749f67c403a9eb8ace0d18e8422" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:54:49.428511 2271 mvcc.cc:205] Tried to move back new op lower bound from 7298466158238957568 to 7298466158013255680. Current Snapshot: MvccSnapshot[applied={T|T < 7298466158238957568}]
I20260619 06:54:49.434382 2272 mvcc.cc:205] Tried to move back new op lower bound from 7298466158238957568 to 7298466158013255680. Current Snapshot: MvccSnapshot[applied={T|T < 7298466158238957568}]
I20260619 06:54:49.442952 1748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:54:49.443329 1748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:54:49.446859 1748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d85a016337064cc28d5e0068156702b0: Sending DeleteTablet for 3 replicas of tablet ba2e06b08875402eaadd3495b106a633
I20260619 06:54:49.448145 1894 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e06b08875402eaadd3495b106a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:35836
I20260619 06:54:49.448199 2172 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e06b08875402eaadd3495b106a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:35814
I20260619 06:54:49.448513 2031 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e06b08875402eaadd3495b106a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:54:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:43866
I20260619 06:54:49.450147 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 1835
I20260619 06:54:49.452689 2280 tablet_replica.cc:330] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: stopping tablet replica
I20260619 06:54:49.453241 2280 raft_consensus.cc:2229] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:54:49.453783 2280 pending_rounds.cc:62] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Trying to abort 2 pending ops.
I20260619 06:54:49.453955 2280 pending_rounds.cc:68] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298466158238957568 op_type: WRITE_OP write_request { tablet_id: "ba2e06b08875402eaadd3495b106a633" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\345\321\323\321\006\"8\n\ntest-admin\022*\n 420fd2e5c41349819c3b8aad48cd82a7\020\001\030\001 \001(\001" signature: "<redacted>""\230;\'\273\242x+\003\253\342\364Lp\274#\024\017\024\346\207 \202\3011\346\237\'q\207Q\333>\203\371\330\307F\223`\201\253vR\370r\234\275\205\024!s\235W\235\013\250\326\200\243p\352\221\264\325" signing_key_seq_num: 0 } } request_id { client_id: "9c15370f5b174f33a157e92034326a3a" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 06:54:49.455010 2281 tablet_replica.cc:330] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: stopping tablet replica
I20260619 06:54:49.455472 2281 raft_consensus.cc:2229] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:54:49.455919 2281 raft_consensus.cc:2258] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:54:49.456866 2280 pending_rounds.cc:68] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298466158013255680 op_type: NO_OP noop_request { }
I20260619 06:54:49.457037 2280 raft_consensus.cc:2866] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 06:54:49.457157 2280 raft_consensus.cc:2258] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:54:49.458225 2281 ts_tablet_manager.cc:1905] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:54:49.461452 2281 ts_tablet_manager.cc:1918] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:54:49.461563 2281 log.cc:1196] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/ba2e06b08875402eaadd3495b106a633
I20260619 06:54:49.461880 2281 ts_tablet_manager.cc:1939] T ba2e06b08875402eaadd3495b106a633 P 9b5e99e435274797826dfb5849f847d9: Deleting consensus metadata
W20260619 06:54:49.462774 2127 connection.cc:535] server connection from 127.22.113.1:37781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:54:49.463014 2280 ts_tablet_manager.cc:1905] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:54:49.464042 1741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:42715 (error 108)
I20260619 06:54:49.465852 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.465441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:42715 (local address 127.0.0.1:35848)
0619 06:54:49.465671 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.465755 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260619 06:54:49.466022 2280 ts_tablet_manager.cc:1918] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:54:49.466116 2280 log.cc:1196] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/ba2e06b08875402eaadd3495b106a633
W20260619 06:54:49.466130 1741 catalog_manager.cc:4584] TS 7f9358d74fff4e278542b54df87b6946 (127.22.113.1:42715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba2e06b08875402eaadd3495b106a633: Network error: Client connection negotiation failed: client connection to 127.22.113.1:42715: connect: Connection refused (error 111)
I20260619 06:54:49.466593 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 1974
I20260619 06:54:49.467497 2280 ts_tablet_manager.cc:1939] T ba2e06b08875402eaadd3495b106a633 P 1fcf3749f67c403a9eb8ace0d18e8422: Deleting consensus metadata
I20260619 06:54:49.469424 1741 catalog_manager.cc:4841] TS 1fcf3749f67c403a9eb8ace0d18e8422 (127.22.113.3:35989): tablet ba2e06b08875402eaadd3495b106a633 (table test-table [id=420fd2e5c41349819c3b8aad48cd82a7]) successfully deleted
I20260619 06:54:49.475224 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 2109
I20260619 06:54:49.479204 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:54:49.478824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45139 (local address 127.0.0.1:43872)
0619 06:54:49.479040 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:54:49.479101 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 06:54:49.483793 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 1722
I20260619 06:54:49.493521 22980 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:54:49.839349 22980 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:54:49.839602 22980 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:54:50.185775 22980 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:54:50.186 UTC [1308] LOG: received smart shutdown request
2026-06-19 06:54:50.190 UTC [1308] LOG: background worker "logical replication launcher" (PID 1317) exited with exit code 1
2026-06-19 06:54:50.190 UTC [1312] LOG: shutting down
2026-06-19 06:54:50.378 UTC [1308] LOG: database system is shut down
2026-06-19T06:54:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49147 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:54:50 dist-test-slave-g8zg krb5kdc[2292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:54:50 dist-test-slave-g8zg krb5kdc[2292](info): set up 2 sockets
Jun 19 06:54:50 dist-test-slave-g8zg krb5kdc[2292](info): commencing operation
krb5kdc: starting...
W20260619 06:54:51.567512 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:54:51 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:54:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangeradmin_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangeradmin_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangerlookup_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangerlookup_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
I20260619 06:54:51.759140 22980 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9tRik3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/postgres -l logfile start
2026-06-19 06:54:54.341 UTC [2322] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:54:54.341 UTC [2322] LOG: listening on IPv4 address "127.22.113.20", port 57059
2026-06-19 06:54:54.345 UTC [2322] LOG: listening on Unix socket "/tmp/.s.PGSQL.57059"
2026-06-19 06:54:54.368 UTC [2325] LOG: database system was shut down at 2026-06-19 06:54:52 UTC
2026-06-19 06:54:54.372 UTC [2322] LOG: database system is ready to accept connections
I20260619 06:54:55.331369 22980 mini_postgres.cc:96] Postgres bound to 57059
2026-06-19 06:54:55.338 UTC [2333] FATAL: database "slave" does not exist
127.22.113.20:57059 - accepting connections
I20260619 06:54:55.339494 22980 mini_ranger.cc:161] Starting Ranger...
I20260619 06:54:55.355458 22980 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:54:55.878715 22980 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:54:55.878824 22980 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin
2026-06-19 06:54:56,168 [I] DB FLAVOR :POSTGRES
2026-06-19 06:54:56,169 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:54:56,169 [I] Checking connection..
2026-06-19 06:54:56,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:54:56,443 [I] Checking connection passed.
2026-06-19 06:54:56,443 [I] --------- Verifying version history table ---------
2026-06-19 06:54:56,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:56,699 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:54:56,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:56,936 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:54:56,937 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:54:56,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:54:57.215 UTC [2397] WARNING: there is no transaction in progress
2026-06-19 06:54:57,222 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:54:57,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:54:57,489 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:54:57,490 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:54:57,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:54:57,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:54:58,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g8zg.c.gcp-upstream.internal','N') ;"
2026-06-19 06:54:58,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:54:58,562 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:54:58,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:54:58,820 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:54:58,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:54:59,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:54:59,341 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:54:59,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:55:00.388 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.406 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.424 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.441 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.462 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.483 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.543 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.551 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.567 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.578 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.589 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.605 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.615 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.622 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.629 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:00.636 UTC [2505] WARNING: there is no transaction in progress
2026-06-19 06:55:01,339 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:55:01,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:55:01,575 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:55:01,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:55:01,825 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:55:01,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:55:02,094 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:55:02,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:55:02,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:55:02,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g8zg.c.gcp-upstream.internal';"
2026-06-19 06:55:02,913 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:55:02,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:55:03,215 [I] DB_PATCHES have already been applied
I20260619 06:55:03.221864 22980 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9tRik3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:55:03.221954 22980 mini_ranger.cc:193] Using host: 127.22.113.20
I20260619 06:55:03.226164 22980 mini_ranger.cc:227] Ranger admin URL: http://127.22.113.20:52623
Jun 19, 2026 6:55:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:55:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:55:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:55:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:55:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:55:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:55:04.381467 22980 mini_ranger.cc:160] Time spent starting Ranger: real 9.042s user 0.002s sys 0.010s
Jun 19, 2026 6:55:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:55:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:55:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:55:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:55:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:55:27.949365 22980 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangerkms_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/rangerkms_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.113.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.113.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.113.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/HTTP_127.22.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:55:28 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852128, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:55:28.033635 22980 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:55:28.043082 22980 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:55:28.979372 22980 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:55:28.979475 22980 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms
I20260619 06:55:28.979494 22980 mini_ranger_kms.cc:227] Using postgres at 127.22.113.20:57059
2026-06-19 06:55:29,162 [I] DB FLAVOR :POSTGRES
2026-06-19 06:55:29,162 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:55:29,162 [I] Checking connection
2026-06-19 06:55:29,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:55:29,415 [I] connection success
2026-06-19 06:55:29,415 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:55:29,415 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:55:29,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:55:29,663 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:55:29,663 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:55:29,664 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:55:29,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.113.20:57059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:55:29,943 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:55:30.479161 22980 mini_ranger_kms.cc:310] Created kms service
I20260619 06:55:30.693830 22980 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:55:30.738951 22980 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:55:30 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852128, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:31.154681 22980 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:55:31.154873 22980 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms:/tmp/dist-test-task9tRik3/build/debug/bin/postgresql.jar:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9tRik3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:55:31.154894 22980 mini_ranger_kms.cc:240] Using host: 127.22.113.20
I20260619 06:55:31.160073 22980 mini_ranger_kms.cc:276] Ranger KMS PID: 2686
I20260619 06:55:31.160187 22980 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.113.20:60153
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:55:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:55:33.540860 22980 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.507s user 0.029s sys 0.135s
I20260619 06:55:33.541034 22980 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:55:33.541126 22980 mini_ranger_kms.cc:401] 127.22.113.20:60153/kms/v1/keys
Jun 19 06:55:33 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:55:34 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
Jun 19 06:55:34 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.113.20@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
Jun 19 06:55:36 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:55:36.846287 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34123
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34123
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:36.980695 2736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:36.980986 2736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:36.981029 2736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:36.985194 2736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:36.985263 2736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:36.985283 2736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:36.985301 2736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:36.985343 2736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:36.990938 2736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.113.20:60153/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34123
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:36.992131 2736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:36.993222 2736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:37.003369 2741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:37.003381 2742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:37.003417 2736 server_base.cc:1034] running on GCE node
W20260619 06:55:37.003369 2744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:37.004555 2736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:37.005785 2736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:37.007016 2736 hybrid_clock.cc:648] HybridClock initialized: now 1781852137006995 us; error 40 us; skew 500 ppm
Jun 19 06:55:37 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852137, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:37.015054 2736 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:37.016208 2736 webserver.cc:460] Webserver started at http://127.22.113.62:35625/ using document root <none> and password file <none>
I20260619 06:55:37.016765 2736 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:37.016824 2736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:37.017011 2736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:55:37 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852137, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:37.068622 2736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "b49d70fea7d942e2997397319983e382"
format_stamp: "Formatted at 2026-06-19 06:55:37 on dist-test-slave-g8zg"
server_key: "e5101ef7a867f57c7d7662e413c9ab02"
server_key_iv: "5079c85d0061b971fefe2221a2fc6226"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:37.069304 2736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "b49d70fea7d942e2997397319983e382"
format_stamp: "Formatted at 2026-06-19 06:55:37 on dist-test-slave-g8zg"
server_key: "e5101ef7a867f57c7d7662e413c9ab02"
server_key_iv: "5079c85d0061b971fefe2221a2fc6226"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:37.073416 2736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:55:37.088541 2751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:37.089975 2736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:37.090126 2736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "b49d70fea7d942e2997397319983e382"
format_stamp: "Formatted at 2026-06-19 06:55:37 on dist-test-slave-g8zg"
server_key: "e5101ef7a867f57c7d7662e413c9ab02"
server_key_iv: "5079c85d0061b971fefe2221a2fc6226"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:37.090278 2736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:37.112423 2736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:37.115554 2736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:37.115774 2736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:37.123991 2736 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:34123
I20260619 06:55:37.124002 2803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:34123 every 8 connection(s)
I20260619 06:55:37.125147 2736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:37.126197 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 2736
I20260619 06:55:37.126375 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:55:37.126698 22980 external_mini_cluster.cc:1428] Setting key cf3a34dd824ddf56575c48ce39e38128
I20260619 06:55:37.129424 2804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:55:37 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:37.137823 2804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Bootstrap starting.
I20260619 06:55:37.141618 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:37.128263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.22.113.62:34123)
0619 06:55:37.128683 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:37.128708 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:37.128745 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:55:37.129399 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:37.129424 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:37.129835 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:37.130207 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:37.131005 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:37.132117 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:37.132988 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:37.133503 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:37.133691 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:37.136376 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:37.136381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:37.136397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:37.136431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:37.138351 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:37.138989 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:37.138995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:37.139001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:37.139129 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:37.139513 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:37.139523 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:37.139525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:37.139990 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:37.140221 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:37.140488 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:55:37.140821 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":149,"threads_started":1}
I20260619 06:55:37.141625 2804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:37.143306 2804 log.cc:824] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:37.145653 2804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: No bootstrap required, opened a new log
I20260619 06:55:37.149107 2804 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:37.149389 2804 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:37.149472 2804 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49d70fea7d942e2997397319983e382, State: Initialized, Role: FOLLOWER
I20260619 06:55:37.149930 2804 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:37.150079 2804 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:37.150166 2804 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:37.150266 2804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:37.151360 2804 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:37.151693 2804 leader_election.cc:302] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49d70fea7d942e2997397319983e382; no voters:
I20260619 06:55:37.152051 2804 leader_election.cc:288] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:55:37.152246 2809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:37.152531 2809 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 LEADER]: Becoming Leader. State: Replica: b49d70fea7d942e2997397319983e382, State: Running, Role: LEADER
I20260619 06:55:37.152933 2809 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:37.153429 2804 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:37.154420 2809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49d70fea7d942e2997397319983e382. Latest consensus state: current_term: 1 leader_uuid: "b49d70fea7d942e2997397319983e382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } } }
I20260619 06:55:37.154553 2809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:37.154961 2818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:37.155218 2810 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b49d70fea7d942e2997397319983e382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } } }
I20260619 06:55:37.155454 2810 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:37.158787 2818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:37.159268 2804 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:55:37.173564 2818 catalog_manager.cc:1347] Generated new cluster ID: a74e23207c034361b3bcf493e92e1fda
I20260619 06:55:37.173753 2818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:37.195282 2818 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:55:37.196813 2818 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:37.209340 2818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Generated new TSK 0
I20260619 06:55:37.210333 2818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:55:37.954841 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:38.094420 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:38.094789 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:38.094861 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:38.101032 2849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:38.101159 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:38.101262 2849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:55:38.109027 2849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:38.110383 2849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:38.111699 2849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:38.124328 2859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:38.125379 2856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:38.125634 2849 server_base.cc:1034] running on GCE node
W20260619 06:55:38.126102 2857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:38.126626 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:38.127408 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:38.128556 2849 hybrid_clock.cc:648] HybridClock initialized: now 1781852138128492 us; error 81 us; skew 500 ppm
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:38.132047 2849 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:55:38.133059 2849 webserver.cc:460] Webserver started at http://127.22.113.1:40149/ using document root <none> and password file <none>
I20260619 06:55:38.133668 2849 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:38.133778 2849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:38.134030 2849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:38.149524 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "7b7f999d926d43448a976623d9c11a1d"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "c69bb4d0a6282bd713c52701e874f5ce"
server_key_iv: "c43a246c86b7b9890b31ff88c747bfa4"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.150347 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "7b7f999d926d43448a976623d9c11a1d"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "c69bb4d0a6282bd713c52701e874f5ce"
server_key_iv: "c43a246c86b7b9890b31ff88c747bfa4"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.156273 2849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 06:55:38.171866 2866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.174019 2849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:55:38.174227 2849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "7b7f999d926d43448a976623d9c11a1d"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "c69bb4d0a6282bd713c52701e874f5ce"
server_key_iv: "c43a246c86b7b9890b31ff88c747bfa4"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.174417 2849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:38.215649 2849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:38.219769 2849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:38.219985 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:38.220775 2849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:38.221982 2849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:38.222041 2849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.222097 2849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:38.222131 2849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.235172 2849 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:38305
I20260619 06:55:38.235242 2979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:38305 every 8 connection(s)
I20260619 06:55:38.236586 2849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:55:38.245703 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 2849
I20260619 06:55:38.245853 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:55:38.246157 22980 external_mini_cluster.cc:1428] Setting key ecb19efa8c0201fd39ef0d2bc25edfe4
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:38.258392 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:38.239130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40063 (local address 127.22.113.62:34123)
0619 06:55:38.240296 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:38.240304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:38.240834 (+ 530us) server_negotiation.cc:407] Connection header received
0619 06:55:38.242003 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:38.242006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:38.242076 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:38.242243 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:38.243910 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.244867 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.246681 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.246990 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.247071 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:38.252921 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:38.252926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:38.252929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:38.252962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:38.255411 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:38.256202 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:38.256206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:38.256208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:38.256325 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:38.256923 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:38.256927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:38.256928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:38.257194 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:38.257286 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:38.258036 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 06:55:38.258188 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":156,"threads_started":1}
I20260619 06:55:38.260136 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:38.239824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.1:40063)
0619 06:55:38.240645 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:38.240705 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:38.241767 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:38.242462 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:38.242472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:38.242871 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:38.243718 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:38.243740 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.245040 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.245043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:38.246509 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:38.246522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.247315 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.247320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:38.247520 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:38.250560 (+ 3040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:38.250585 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:38.252687 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:38.255589 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:38.255596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:38.255627 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:38.256048 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:38.256485 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:38.256490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:38.256493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:38.256795 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:38.257344 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:38.257352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:38.257739 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 06:55:38.258650 (+ 911us) client_negotiation.cc:240] Negotiation successful
0619 06:55:38.259045 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":361,"threads_started":1}
I20260619 06:55:38.261922 2982 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:38.262185 2982 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:38.262809 2982 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:55:38.264868 2768 ts_manager.cc:194] Registered new tserver with Master: 7b7f999d926d43448a976623d9c11a1d (127.22.113.1:38305)
I20260619 06:55:38.266659 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:40063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:55:38.294277 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:55:38.469967 2988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:38.470424 2988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:38.470534 2988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:38.475819 2988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:38.475917 2988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:38.476016 2988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:55:38.482928 2988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:38.484697 2988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:38.486231 2988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:38.500507 2995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:38.500938 2997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:38.501477 2994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:38.502828 2988 server_base.cc:1034] running on GCE node
I20260619 06:55:38.503594 2988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:38.504572 2988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:38.505800 2988 hybrid_clock.cc:648] HybridClock initialized: now 1781852138505766 us; error 49 us; skew 500 ppm
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:38.509852 2988 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:55:38.511191 2988 webserver.cc:460] Webserver started at http://127.22.113.2:44715/ using document root <none> and password file <none>
I20260619 06:55:38.512058 2988 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:38.512156 2988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:38.512424 2988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:38.538193 2988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "e1b3d875d5ac4ae69e6e565f5c12c125"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "9e50e527ddcc8e50b794cfddf592b1af"
server_key_iv: "31575e6c510e7f02ab570c75046f4613"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.540506 2988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "e1b3d875d5ac4ae69e6e565f5c12c125"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "9e50e527ddcc8e50b794cfddf592b1af"
server_key_iv: "31575e6c510e7f02ab570c75046f4613"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.548915 2988 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 06:55:38.568507 3004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.570235 2988 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:55:38.570385 2988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "e1b3d875d5ac4ae69e6e565f5c12c125"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "9e50e527ddcc8e50b794cfddf592b1af"
server_key_iv: "31575e6c510e7f02ab570c75046f4613"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.570565 2988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:38.602686 2988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:38.606073 2988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:38.606287 2988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:38.606967 2988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:38.608098 2988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:38.608161 2988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.608223 2988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:38.608255 2988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.618690 2988 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:36409
I20260619 06:55:38.619720 2988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:55:38.628983 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 2988
I20260619 06:55:38.629122 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:38.629423 22980 external_mini_cluster.cc:1428] Setting key b47acf0df7e6a47a9dbee5f7dfb89b85
I20260619 06:55:38.630121 3117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:36409 every 8 connection(s)
I20260619 06:55:38.635331 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:38.621749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:52515 (local address 127.22.113.62:34123)
0619 06:55:38.621947 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:38.621953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:38.622736 (+ 783us) server_negotiation.cc:407] Connection header received
0619 06:55:38.623827 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:38.623830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:38.623893 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:38.624001 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:38.625035 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.625579 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.626680 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.626904 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.627012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:38.629773 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:38.629777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:38.629781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:38.629812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:38.631507 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:38.633687 (+ 2180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:38.633693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:38.633695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:38.633787 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:38.634217 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:38.634223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:38.634228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:38.634413 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:38.634478 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:38.635016 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 06:55:38.635141 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:55:38.636173 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:38.622111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.2:52515)
0619 06:55:38.622591 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:38.622642 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:38.623614 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:38.624176 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:38.624182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:38.624438 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:38.624886 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:38.624898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.625702 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.625705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:38.626551 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:38.626563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:38.627060 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:38.627064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:38.627225 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:38.627997 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:38.628018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:38.629583 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:38.633135 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:38.633143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:38.633160 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:38.633558 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:38.633902 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:38.633907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:38.633912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:38.634100 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:38.634545 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:38.634550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:38.634774 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:55:38.634966 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:55:38.635286 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":144,"threads_started":1}
I20260619 06:55:38.637653 3118 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:38.637928 3118 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:38.638569 3118 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:38.639875 2768 ts_manager.cc:194] Registered new tserver with Master: e1b3d875d5ac4ae69e6e565f5c12c125 (127.22.113.2:36409)
I20260619 06:55:38.640719 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:52515
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:55:38.674115 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:38.854701 3123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:38.855052 3123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:38.855111 3123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:38.861214 3123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:38.861312 3123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:38.861395 3123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:55:38.869174 3123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:38.870595 3123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:38.872074 3123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:38.885092 3129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:38.885550 3128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:38.885746 3131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:38.886781 3123 server_base.cc:1034] running on GCE node
I20260619 06:55:38.887310 3123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:38.888037 3123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:38.903507 3123 hybrid_clock.cc:648] HybridClock initialized: now 1781852138903370 us; error 145 us; skew 500 ppm
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:38.907338 3123 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:55:38.908651 3123 webserver.cc:460] Webserver started at http://127.22.113.3:34561/ using document root <none> and password file <none>
I20260619 06:55:38.909381 3123 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:38.909471 3123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:38.909721 3123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:55:38 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:38.937942 3123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "6eb1b29dc1004f9fb5001ad252281d3e"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "3c081a33283132090b6a2df86aef083f"
server_key_iv: "00e083a2fb31a31d5b02081d0e8e02b0"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.938658 3123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "6eb1b29dc1004f9fb5001ad252281d3e"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "3c081a33283132090b6a2df86aef083f"
server_key_iv: "00e083a2fb31a31d5b02081d0e8e02b0"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.944763 3123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 06:55:38.968124 3138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.969825 3123 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:55:38.969954 3123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6eb1b29dc1004f9fb5001ad252281d3e"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "3c081a33283132090b6a2df86aef083f"
server_key_iv: "00e083a2fb31a31d5b02081d0e8e02b0"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:38.970079 3123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:38.985896 3123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:38.989565 3123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:38.989814 3123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:38.990479 3123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:38.991539 3123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:38.991637 3123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:38.991742 3123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:38.991814 3123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.004561 3123 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40031
I20260619 06:55:39.005720 3123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:55:39.007668 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3123
I20260619 06:55:39.007838 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:55:39.008193 22980 external_mini_cluster.cc:1428] Setting key 16223019021b1823214007d240c52215
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852138, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:39.006588 3251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40031 every 8 connection(s)
I20260619 06:55:39.029277 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.010479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:44815 (local address 127.22.113.62:34123)
0619 06:55:39.010605 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:39.010609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:39.015892 (+ 5283us) server_negotiation.cc:407] Connection header received
0619 06:55:39.017148 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:39.017152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:39.017210 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:39.017264 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:39.018880 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.019700 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.021346 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.021687 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.021765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.024856 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:39.024860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:39.024863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:39.024888 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:39.026602 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.027193 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.027197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.027199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.027315 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.028024 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.028027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.028029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.028234 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:39.028303 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:39.028950 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 06:55:39.029079 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:55:39.031097 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.009419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.3:44815)
0619 06:55:39.015688 (+ 6269us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:39.015763 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:39.016891 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:39.017683 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:39.017693 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:39.018059 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:39.018702 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.018720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.019873 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.019876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.021207 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.021218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.021971 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.021974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.022157 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.022872 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:39.022893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:39.024649 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:39.026739 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.026744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.026773 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.027082 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.027725 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.027728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.027730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.027912 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.028380 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:39.028386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:39.028676 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 06:55:39.029750 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0619 06:55:39.030087 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6019,"thread_start_us":141,"threads_started":1}
I20260619 06:55:39.032747 3252 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:39.033003 3252 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.033579 3252 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:39.034765 2768 ts_manager.cc:194] Registered new tserver with Master: 6eb1b29dc1004f9fb5001ad252281d3e (127.22.113.3:40031)
I20260619 06:55:39.035589 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:44815
I20260619 06:55:39.045974 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:55:39.046773 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 2849
I20260619 06:55:39.055281 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 2988
I20260619 06:55:39.062057 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3123
I20260619 06:55:39.070433 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 2736
I20260619 06:55:39.076956 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34123
--webserver_interface=127.22.113.62
--webserver_port=35625
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34123
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:39.189170 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:39.189486 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:39.189559 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:39.193709 3255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:39.193792 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:39.193825 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:39.193864 3255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:39.193894 3255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:39.199216 3255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34123
--ranger_config_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:39.200242 3255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:39.201201 3255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:39.210027 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:39.209971 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:39.210049 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:39.210500 3255 server_base.cc:1034] running on GCE node
I20260619 06:55:39.211033 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:39.212136 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:39.213313 3255 hybrid_clock.cc:648] HybridClock initialized: now 1781852139213273 us; error 55 us; skew 500 ppm
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:39.216408 3255 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:39.217335 3255 webserver.cc:460] Webserver started at http://127.22.113.62:35625/ using document root <none> and password file <none>
I20260619 06:55:39.217919 3255 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:39.218019 3255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:39.273291 3255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:39.275753 3271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.277163 3255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:39.277331 3255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "b49d70fea7d942e2997397319983e382"
format_stamp: "Formatted at 2026-06-19 06:55:37 on dist-test-slave-g8zg"
server_key: "e5101ef7a867f57c7d7662e413c9ab02"
server_key_iv: "5079c85d0061b971fefe2221a2fc6226"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:39.277884 3255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:39.296990 3255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:39.300051 3255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:39.300511 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:39.308161 3255 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:34123
I20260619 06:55:39.308192 3323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:34123 every 8 connection(s)
I20260619 06:55:39.309278 3255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:39.313539 3324 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:55:39.314913 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3255
I20260619 06:55:39.315222 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Bootstrap starting.
I20260619 06:55:39.315572 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:38305
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:34123
--webserver_port=40149
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:39.337275 3324 log.cc:824] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:39.345842 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:55:39.346297 3324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Bootstrap complete.
I20260619 06:55:39.351742 3324 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:39.352247 3324 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49d70fea7d942e2997397319983e382, State: Initialized, Role: FOLLOWER
I20260619 06:55:39.352809 3324 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:39.352957 3324 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:39.353029 3324 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:39.353127 3324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:55:39.355568 3324 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:39.355917 3324 leader_election.cc:302] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49d70fea7d942e2997397319983e382; no voters:
I20260619 06:55:39.356295 3324 leader_election.cc:288] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:55:39.356544 3328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:55:39.356911 3328 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [term 2 LEADER]: Becoming Leader. State: Replica: b49d70fea7d942e2997397319983e382, State: Running, Role: LEADER
I20260619 06:55:39.357347 3328 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } }
I20260619 06:55:39.357700 3324 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:39.359083 3330 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49d70fea7d942e2997397319983e382. Latest consensus state: current_term: 2 leader_uuid: "b49d70fea7d942e2997397319983e382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } } }
I20260619 06:55:39.359261 3330 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:39.359551 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b49d70fea7d942e2997397319983e382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49d70fea7d942e2997397319983e382" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34123 } } }
I20260619 06:55:39.359643 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:39.360049 3338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:39.360172 3324 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:55:39.367178 3338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:39.368023 3338 catalog_manager.cc:1259] Loaded cluster ID: a74e23207c034361b3bcf493e92e1fda
I20260619 06:55:39.368103 3338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:39.370836 3338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:39.371706 3338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Loaded TSK: 0
I20260619 06:55:39.373041 3338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:55:39.492257 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:39.492700 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:39.492784 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:39.499231 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:39.499361 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:39.499462 3326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:55:39.508137 3326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:38305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=40149
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:39.509809 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:39.511850 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:39.526081 3362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:39.526163 3364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:39.526149 3326 server_base.cc:1034] running on GCE node
W20260619 06:55:39.526090 3361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:39.527053 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:39.527985 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:39.529208 3326 hybrid_clock.cc:648] HybridClock initialized: now 1781852139529184 us; error 43 us; skew 500 ppm
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:39.533206 3326 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:55:39.534621 3326 webserver.cc:460] Webserver started at http://127.22.113.1:40149/ using document root <none> and password file <none>
I20260619 06:55:39.535498 3326 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:39.535637 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:39.556435 3326 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:55:39.560206 3372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.561848 3326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:55:39.562011 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "7b7f999d926d43448a976623d9c11a1d"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "c69bb4d0a6282bd713c52701e874f5ce"
server_key_iv: "c43a246c86b7b9890b31ff88c747bfa4"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:39.562793 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:39.580968 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:39.584677 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:39.584895 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:39.585693 3326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:39.586884 3326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:39.586948 3326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.587002 3326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:39.587035 3326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.598551 3326 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:38305
I20260619 06:55:39.598721 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:38305 every 8 connection(s)
I20260619 06:55:39.599954 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:55:39.606506 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3326
I20260619 06:55:39.607137 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:36409
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:34123
--webserver_port=44715
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:39.640030 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.602949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:38769 (local address 127.22.113.62:34123)
0619 06:55:39.613696 (+ 10747us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:39.613724 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:39.613788 (+ 64us) server_negotiation.cc:407] Connection header received
0619 06:55:39.614965 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:39.614989 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:39.615534 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:39.615936 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:39.617486 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.620569 (+ 3083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.622773 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.623715 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.623933 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.630527 (+ 6594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:39.630536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:39.630557 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:39.630593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:39.632875 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.635303 (+ 2428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.635311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.635316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.635467 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.635865 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.635868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.635870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.636402 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:39.636527 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:39.638714 (+ 2187us) server_negotiation.cc:299] Negotiation successful
0619 06:55:39.639022 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10533,"thread_start_us":201,"threads_started":1}
I20260619 06:55:39.641223 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.602969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.1:38769)
0619 06:55:39.612856 (+ 9887us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:39.612932 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:39.614321 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:39.615999 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:39.616008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:39.616431 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:39.617166 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.617190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.620829 (+ 3639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.620834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.622567 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.622583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.625580 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.625586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.625849 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.627724 (+ 1875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:39.627749 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:39.630248 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:39.634723 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.634731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.634749 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.635119 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.635574 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.635577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.635579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.635759 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.636590 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:39.636597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:39.636944 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 06:55:39.637157 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 06:55:39.637557 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9569,"thread_start_us":224,"threads_started":1}
I20260619 06:55:39.644896 3486 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:39.645159 3486 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.645784 3486 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
W20260619 06:55:39.648362 3486 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:34123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:55:39.648487 3486 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.648705 3486 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:39.649776 3486 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.649959 3486 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
W20260619 06:55:39.650741 3486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:55:39.774374 3490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:39.774892 3490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:39.774991 3490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:39.779819 3490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:39.779927 3490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:39.780028 3490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:55:39.787813 3490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:36409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=44715
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:39.789597 3490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:39.790844 3490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:39.805639 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:39.806588 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:39.806895 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:39.807279 3490 server_base.cc:1034] running on GCE node
I20260619 06:55:39.807888 3490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:39.808820 3490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:39.816311 3490 hybrid_clock.cc:648] HybridClock initialized: now 1781852139816266 us; error 83 us; skew 500 ppm
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:39.820870 3490 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:55:39.822361 3490 webserver.cc:460] Webserver started at http://127.22.113.2:44715/ using document root <none> and password file <none>
I20260619 06:55:39.823385 3490 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:39.823575 3490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:39.854717 3490 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:55:39.858043 3506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.860368 3490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:55:39.860525 3490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "e1b3d875d5ac4ae69e6e565f5c12c125"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "9e50e527ddcc8e50b794cfddf592b1af"
server_key_iv: "31575e6c510e7f02ab570c75046f4613"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:39.861462 3490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:39.883051 3490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:39.886657 3490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:39.886930 3490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:39.887764 3490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:39.888844 3490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:39.888917 3490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.888993 3490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:39.889035 3490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:39.902755 3490 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:36409
I20260619 06:55:39.904361 3490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:55:39.907341 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3490
I20260619 06:55:39.907949 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.113.20:60153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:40031
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:34123
--webserver_port=34561
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:39.908622 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:36409 every 8 connection(s)
Jun 19 06:55:39 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852139, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:39.928475 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.911524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:34083 (local address 127.22.113.62:34123)
0619 06:55:39.911722 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:39.911730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:39.912821 (+ 1091us) server_negotiation.cc:407] Connection header received
0619 06:55:39.914638 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:39.914643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:39.914728 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:39.914865 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:39.916854 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.917719 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.920039 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.920471 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.920591 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.923988 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:39.923992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:39.923995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:39.924025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:39.925728 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.926440 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.926446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.926448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.926559 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:39.926971 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:39.926975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:39.926977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:39.927232 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:39.927347 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:39.928129 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 06:55:39.928286 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:55:39.929720 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:39.911828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.2:34083)
0619 06:55:39.912591 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:39.912670 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:39.914303 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:39.915141 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:39.915156 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:39.915686 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:39.916621 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.916654 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.917899 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.917905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.919867 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:39.919881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:39.920676 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:39.920683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:39.921039 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:39.921760 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:39.921788 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:39.923751 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:39.925866 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.925873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.925894 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.926289 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.926684 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:39.926686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:39.926688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:39.926864 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:39.927412 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:39.927420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:39.927780 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 06:55:39.928002 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 06:55:39.928395 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":201,"threads_started":1}
I20260619 06:55:39.931181 3621 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:39.931562 3621 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.932777 3621 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
W20260619 06:55:39.934867 3621 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:34123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:55:39.935022 3621 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.935361 3621 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:39.936432 3621 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:39.936635 3621 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
W20260619 06:55:39.937394 3621 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:55:40.069670 3623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:40.070190 3623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:40.070358 3623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:40.076418 3623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:40.076530 3623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:40.076642 3623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:55:40.084824 3623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.113.20:60153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:40031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=34561
--tserver_master_addrs=127.22.113.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:40.086602 3623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:40.088235 3623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:40.102030 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:40.102030 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:40.102537 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:40.103267 3623 server_base.cc:1034] running on GCE node
I20260619 06:55:40.103852 3623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:40.104835 3623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:40.112262 3623 hybrid_clock.cc:648] HybridClock initialized: now 1781852140112207 us; error 61 us; skew 500 ppm
Jun 19 06:55:40 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:40.116808 3623 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:55:40.118119 3623 webserver.cc:460] Webserver started at http://127.22.113.3:34561/ using document root <none> and password file <none>
I20260619 06:55:40.118950 3623 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:40.119064 3623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:55:40 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:40.140173 3623 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:55:40.144393 3640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:40.146551 3623 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:55:40.146754 3623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6eb1b29dc1004f9fb5001ad252281d3e"
format_stamp: "Formatted at 2026-06-19 06:55:38 on dist-test-slave-g8zg"
server_key: "3c081a33283132090b6a2df86aef083f"
server_key_iv: "00e083a2fb31a31d5b02081d0e8e02b0"
server_key_version: "kuduclusterkey@0"
I20260619 06:55:40.147807 3623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:40.168442 3623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:40.173157 3623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:40.173430 3623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:40.174369 3623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:40.175647 3623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:40.175717 3623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:40.175783 3623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:40.175818 3623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:40.189812 3623 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40031
I20260619 06:55:40.190097 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40031 every 8 connection(s)
I20260619 06:55:40.191401 3623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:55:40.200282 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3623
Jun 19 06:55:40 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:40.210144 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.193839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:43509 (local address 127.22.113.62:34123)
0619 06:55:40.193992 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.193997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.195420 (+ 1423us) server_negotiation.cc:407] Connection header received
0619 06:55:40.196518 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.196521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.196577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.196675 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:40.197902 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.198442 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.199805 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.200236 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.200415 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.203801 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:40.203806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:40.203809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:40.203839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:40.207319 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.208249 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.208252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.208254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.208347 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.208741 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.208744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.208746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.209005 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:40.209114 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.209811 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.209970 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:55:40.211843 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.194503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34123 (local address 127.22.113.3:43509)
0619 06:55:40.195169 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:40.195222 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:40.196287 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:40.196899 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:40.196907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:40.197230 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:40.197746 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.197766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.198566 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.198570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.199633 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.199644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.200826 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.200832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.201050 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.201803 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:40.201824 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:40.203579 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:40.207694 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:40.207703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:40.207724 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:40.208124 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:40.208453 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:40.208456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:40.208458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:40.208636 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:40.209184 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:40.209191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:40.209518 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:55:40.210394 (+ 876us) client_negotiation.cc:240] Negotiation successful
0619 06:55:40.210771 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":165,"threads_started":1}
I20260619 06:55:40.213164 3761 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34123
I20260619 06:55:40.213446 3761 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:40.213865 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.201794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.22.113.62:34123)
0619 06:55:40.202154 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.202167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.202528 (+ 361us) server_negotiation.cc:407] Connection header received
0619 06:55:40.202760 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.202764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.202822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.202893 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:40.204200 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.205077 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.206392 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.206747 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.206832 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.208962 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:40.208966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:40.209010 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:40.209059 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:40.211305 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.211801 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.211805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.211807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.211892 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.212499 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.212503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.212505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.212750 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:40.212987 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.213553 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.213709 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":140,"threads_started":1}
I20260619 06:55:40.214080 3761 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:40.216123 3288 ts_manager.cc:194] Registered new tserver with Master: 6eb1b29dc1004f9fb5001ad252281d3e (127.22.113.3:40031)
I20260619 06:55:40.218155 3288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:43509
Jun 19 06:55:40 dist-test-slave-g8zg krb5kdc[2292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:40.651005 3486 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:40.651778 3486 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:40.652621 3288 ts_manager.cc:194] Registered new tserver with Master: 7b7f999d926d43448a976623d9c11a1d (127.22.113.1:38305)
I20260619 06:55:40.653523 3288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:38769
I20260619 06:55:40.937675 3621 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:40.938066 3621 heartbeater.cc:511] Master 127.22.113.62:34123 requested a full tablet report, sending...
I20260619 06:55:40.938800 3288 ts_manager.cc:194] Registered new tserver with Master: e1b3d875d5ac4ae69e6e565f5c12c125 (127.22.113.2:36409)
I20260619 06:55:40.939468 3288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:34083
I20260619 06:55:40.947214 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:55:40.957005 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.948935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.22.113.62:34123)
0619 06:55:40.949207 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.949211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.949223 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:55:40.949349 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.949352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.949405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.949501 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:40.950178 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.950726 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.951642 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.951919 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.952043 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.953607 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:40.953611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:40.953614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:40.953638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:40.955326 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.955754 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.955759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.955761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.955847 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:40.956114 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:40.956119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:40.956121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:40.956306 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:40.956447 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.956669 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.956797 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260619 06:55:40.961417 3288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:55:40.966043 3288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:55:40.990903 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.982915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36409 (local address 127.0.0.1:54280)
0619 06:55:40.984390 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:40.984400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:40.984499 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:40.985380 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:40.985387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:40.985425 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:40.985790 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.985798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.987164 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.987168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.988815 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.988826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.990139 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.990143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.990252 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.990302 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:55:40.990581 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 06:55:40.990715 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":98,"spinlock_wait_cycles":3200,"thread_start_us":125,"threads_started":1}
I20260619 06:55:40.991401 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.983229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.22.113.2:36409)
0619 06:55:40.983684 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.983692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.984488 (+ 796us) server_negotiation.cc:407] Connection header received
0619 06:55:40.984804 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.984809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.985028 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.985182 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:40.985934 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.987034 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.988954 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.989808 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.989907 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.990079 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.991115 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.991228 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20260619 06:55:40.991932 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.981743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:40031 (local address 127.0.0.1:52724)
0619 06:55:40.984021 (+ 2278us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:40.984044 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:40.984159 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:40.985403 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:40.985405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:40.985430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:40.985782 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.985798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.988009 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.988012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.989105 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.989113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.990946 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.990950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.991701 (+ 751us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.991721 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:55:40.991770 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 06:55:40.991832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"mutex_wait_us":101,"spinlock_wait_cycles":7424,"thread_start_us":104,"threads_started":1}
I20260619 06:55:40.992619 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.982164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.22.113.3:40031)
0619 06:55:40.984821 (+ 2657us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.984829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.984853 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:55:40.984962 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.984968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.985248 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.985431 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:40.985958 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.987855 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.990008 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.990814 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.992230 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.992337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.992377 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.992483 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":93,"threads_started":1}
I20260619 06:55:40.994874 3549 tablet_service.cc:1467] Processing CreateTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd (DEFAULT_TABLE table=test-table [id=a68a27c07597487f9856b58f1eb8a8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:40.995275 3690 tablet_service.cc:1467] Processing CreateTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd (DEFAULT_TABLE table=test-table [id=a68a27c07597487f9856b58f1eb8a8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:40.996017 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.979983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.22.113.1:38305)
0619 06:55:40.980404 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:40.980411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:40.983884 (+ 3473us) server_negotiation.cc:407] Connection header received
0619 06:55:40.990519 (+ 6635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:40.990526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:40.990818 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:40.990957 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:40.991516 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.992677 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.994243 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.995089 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.995184 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.995363 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:40.995729 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:55:40.995844 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":121,"threads_started":1}
I20260619 06:55:40.996258 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ead86afcbaf4e1d998b537a079ddbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:40.996438 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:40.979106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:38305 (local address 127.0.0.1:58904)
0619 06:55:40.983641 (+ 4535us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:40.983725 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:40.987871 (+ 4146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:40.991184 (+ 3313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:40.991187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:40.991208 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:40.991375 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.991381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.992827 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.992830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.994094 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:40.994103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:40.995463 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:40.995467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:40.995576 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:40.995600 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:55:40.996192 (+ 592us) client_negotiation.cc:240] Negotiation successful
0619 06:55:40.996291 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4252,"thread_start_us":111,"threads_started":1}
I20260619 06:55:40.996686 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ead86afcbaf4e1d998b537a079ddbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:41.002740 3419 tablet_service.cc:1467] Processing CreateTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd (DEFAULT_TABLE table=test-table [id=a68a27c07597487f9856b58f1eb8a8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:41.003846 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ead86afcbaf4e1d998b537a079ddbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:41.006031 3787 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Bootstrap starting.
I20260619 06:55:41.008416 3787 tablet_bootstrap.cc:654] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:41.009584 3787 log.cc:824] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:55:41 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:41.016705 3789 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Bootstrap starting.
I20260619 06:55:41.019318 3789 tablet_bootstrap.cc:654] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:41.020318 3789 log.cc:824] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:41.020736 3790 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Bootstrap starting.
I20260619 06:55:41.022924 3790 tablet_bootstrap.cc:654] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:41.023761 3790 log.cc:824] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:41.026497 3787 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: No bootstrap required, opened a new log
I20260619 06:55:41.026741 3787 ts_tablet_manager.cc:1397] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260619 06:55:41.030926 3787 raft_consensus.cc:348] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.031234 3787 raft_consensus.cc:374] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:41.031854 3787 raft_consensus.cc:729] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b3d875d5ac4ae69e6e565f5c12c125, State: Initialized, Role: FOLLOWER
I20260619 06:55:41.032361 3787 consensus_queue.cc:260] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.033347 3621 heartbeater.cc:503] Master 127.22.113.62:34123 was elected leader, sending a full tablet report...
I20260619 06:55:41.033576 3787 ts_tablet_manager.cc:1428] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:55:41.035465 3790 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: No bootstrap required, opened a new log
I20260619 06:55:41.035696 3790 ts_tablet_manager.cc:1397] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260619 06:55:41.040406 3790 raft_consensus.cc:348] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.040720 3790 raft_consensus.cc:374] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:41.040802 3790 raft_consensus.cc:729] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb1b29dc1004f9fb5001ad252281d3e, State: Initialized, Role: FOLLOWER
I20260619 06:55:41.041333 3790 consensus_queue.cc:260] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.042169 3761 heartbeater.cc:503] Master 127.22.113.62:34123 was elected leader, sending a full tablet report...
I20260619 06:55:41.043395 3789 tablet_bootstrap.cc:492] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: No bootstrap required, opened a new log
I20260619 06:55:41.043668 3789 ts_tablet_manager.cc:1397] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20260619 06:55:41.043933 3790 ts_tablet_manager.cc:1428] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:55:41.047793 3789 raft_consensus.cc:348] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.048182 3789 raft_consensus.cc:374] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:41.048328 3789 raft_consensus.cc:729] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7f999d926d43448a976623d9c11a1d, State: Initialized, Role: FOLLOWER
I20260619 06:55:41.048883 3789 consensus_queue.cc:260] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.050091 3789 ts_tablet_manager.cc:1428] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:55:41.050494 3486 heartbeater.cc:503] Master 127.22.113.62:34123 was elected leader, sending a full tablet report...
W20260619 06:55:41.102308 3487 tablet.cc:2369] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:55:41.161731 3622 tablet.cc:2369] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:55:41.193750 3762 tablet.cc:2369] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:55:41 dist-test-slave-g8zg krb5kdc[2292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852140, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for HTTP/127.22.113.20@KRBTEST.COM
I20260619 06:55:41.331983 3795 raft_consensus.cc:482] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:41.332330 3795 raft_consensus.cc:504] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.334208 3795 leader_election.cc:288] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eb1b29dc1004f9fb5001ad252281d3e (127.22.113.3:40031), e1b3d875d5ac4ae69e6e565f5c12c125 (127.22.113.2:36409)
I20260619 06:55:41.339922 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.334595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:52531 (local address 127.22.113.2:36409)
0619 06:55:41.334757 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:41.334762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:41.335179 (+ 417us) server_negotiation.cc:407] Connection header received
0619 06:55:41.335464 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:41.335468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:41.335529 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:41.335587 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:41.336098 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.336829 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.338483 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.339098 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.339200 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:41.339294 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:41.339670 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 06:55:41.339758 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:55:41.341688 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.334445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36409 (local address 127.22.113.1:52531)
0619 06:55:41.335080 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:41.335098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:41.335315 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:41.335756 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:41.335758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:41.335777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:41.335961 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:41.335971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.336990 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.336995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:41.338342 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:41.338353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.339393 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.339396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:41.339514 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:41.339542 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:55:41.340336 (+ 794us) client_negotiation.cc:240] Negotiation successful
0619 06:55:41.340419 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":136,"threads_started":1}
I20260619 06:55:41.342952 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ead86afcbaf4e1d998b537a079ddbfd" candidate_uuid: "7b7f999d926d43448a976623d9c11a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" is_pre_election: true
I20260619 06:55:41.343464 3563 raft_consensus.cc:2454] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7f999d926d43448a976623d9c11a1d in term 0.
I20260619 06:55:41.344403 3376 leader_election.cc:302] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7f999d926d43448a976623d9c11a1d, e1b3d875d5ac4ae69e6e565f5c12c125; no voters:
I20260619 06:55:41.344851 3795 raft_consensus.cc:2781] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:55:41.344938 3795 raft_consensus.cc:482] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:55:41.344980 3795 raft_consensus.cc:3037] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:41.346633 3795 raft_consensus.cc:504] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.347121 3795 leader_election.cc:288] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [CANDIDATE]: Term 1 election: Requested vote from peers 6eb1b29dc1004f9fb5001ad252281d3e (127.22.113.3:40031), e1b3d875d5ac4ae69e6e565f5c12c125 (127.22.113.2:36409)
I20260619 06:55:41.348094 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ead86afcbaf4e1d998b537a079ddbfd" candidate_uuid: "7b7f999d926d43448a976623d9c11a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125"
I20260619 06:55:41.348269 3563 raft_consensus.cc:3037] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:41.349828 3563 raft_consensus.cc:2454] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7f999d926d43448a976623d9c11a1d in term 1.
I20260619 06:55:41.350365 3376 leader_election.cc:302] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7f999d926d43448a976623d9c11a1d, e1b3d875d5ac4ae69e6e565f5c12c125; no voters:
I20260619 06:55:41.350637 3795 raft_consensus.cc:2781] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:41.350906 3795 raft_consensus.cc:686] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [term 1 LEADER]: Becoming Leader. State: Replica: 7b7f999d926d43448a976623d9c11a1d, State: Running, Role: LEADER
I20260619 06:55:41.351425 3795 consensus_queue.cc:237] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } }
I20260619 06:55:41.356205 3282 catalog_manager.cc:5495] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7f999d926d43448a976623d9c11a1d (127.22.113.1). New cstate: current_term: 1 leader_uuid: "7b7f999d926d43448a976623d9c11a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7f999d926d43448a976623d9c11a1d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 38305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 } health_report { overall_health: UNKNOWN } } }
I20260619 06:55:41.358616 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.334638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:56805 (local address 127.22.113.3:40031)
0619 06:55:41.334892 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:41.334899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:41.342185 (+ 7286us) server_negotiation.cc:407] Connection header received
0619 06:55:41.342404 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:41.342407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:41.342471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:41.342526 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:41.342936 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.343652 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.356641 (+ 12989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.357220 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.357318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:41.357375 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:41.358379 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0619 06:55:41.358480 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 06:55:41.359890 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.334445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:40031 (local address 127.22.113.1:56805)
0619 06:55:41.342110 (+ 7665us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:41.342125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:41.342269 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:41.342700 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:41.342703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:41.342720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:41.342792 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:41.342797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.353929 (+ 11132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.353933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:41.355136 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:41.356469 (+ 1333us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.358129 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.358133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:41.358223 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:41.358248 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:55:41.359639 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0619 06:55:41.359731 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7595,"mutex_wait_us":241,"thread_start_us":94,"threads_started":1}
I20260619 06:55:41.361024 3715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ead86afcbaf4e1d998b537a079ddbfd" candidate_uuid: "7b7f999d926d43448a976623d9c11a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" is_pre_election: true
I20260619 06:55:41.361337 3715 raft_consensus.cc:2454] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7f999d926d43448a976623d9c11a1d in term 0.
I20260619 06:55:41.361660 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ead86afcbaf4e1d998b537a079ddbfd" candidate_uuid: "7b7f999d926d43448a976623d9c11a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eb1b29dc1004f9fb5001ad252281d3e"
I20260619 06:55:41.361759 3714 raft_consensus.cc:3037] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:41.363073 3714 raft_consensus.cc:2454] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7f999d926d43448a976623d9c11a1d in term 1.
I20260619 06:55:41.377249 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.372153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.22.113.1:38305)
0619 06:55:41.372270 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:41.372274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:41.372287 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:55:41.372365 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:41.372368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:41.372445 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:41.372547 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:55:41.372930 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.373727 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.374969 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:41.375349 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:41.375465 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:41.375896 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:55:41.376479 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:55:41.376597 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:41.377014 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 06:55:41.377083 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:55:41.388276 3714 raft_consensus.cc:1261] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 1 FOLLOWER]: Refusing update from remote peer 7b7f999d926d43448a976623d9c11a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:41.388581 3563 raft_consensus.cc:1261] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 1 FOLLOWER]: Refusing update from remote peer 7b7f999d926d43448a976623d9c11a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:41.389159 3800 consensus_queue.cc:1035] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eb1b29dc1004f9fb5001ad252281d3e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:41.389364 3795 consensus_queue.cc:1035] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b3d875d5ac4ae69e6e565f5c12c125" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:41.401185 3803 mvcc.cc:205] Tried to move back new op lower bound from 7298466371112460288 to 7298466370979627008. Current Snapshot: MvccSnapshot[applied={T|T < 7298466371112460288}]
I20260619 06:55:41.410406 3805 mvcc.cc:205] Tried to move back new op lower bound from 7298466371112460288 to 7298466370979627008. Current Snapshot: MvccSnapshot[applied={T|T < 7298466371112460288}]
I20260619 06:55:41.420207 3288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:41.420465 3288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:41.421047 3804 mvcc.cc:205] Tried to move back new op lower bound from 7298466371112460288 to 7298466370979627008. Current Snapshot: MvccSnapshot[applied={T|T < 7298466371112460288}]
I20260619 06:55:41.424033 3288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b49d70fea7d942e2997397319983e382: Sending DeleteTablet for 3 replicas of tablet 0ead86afcbaf4e1d998b537a079ddbfd
I20260619 06:55:41.425297 3419 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:41 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:58904
I20260619 06:55:41.425371 3690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:41 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:52724
I20260619 06:55:41.425856 3549 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ead86afcbaf4e1d998b537a079ddbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:41 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:54280
I20260619 06:55:41.426565 3812 tablet_replica.cc:330] T 0ead86afcbaf4e1d998b537a079ddbfd P 7b7f999d926d43448a976623d9c11a1d: stopping tablet replica
I20260619 06:55:41.426733 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3326
I20260619 06:55:41.426792 3814 tablet_replica.cc:330] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: stopping tablet replica
I20260619 06:55:41.427124 3814 raft_consensus.cc:2229] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:55:41.427688 3814 raft_consensus.cc:2258] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:41.436096 3813 tablet_replica.cc:330] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: stopping tablet replica
I20260619 06:55:41.436695 3813 raft_consensus.cc:2229] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:55:41.437208 3813 raft_consensus.cc:2258] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:41.437219 3814 ts_tablet_manager.cc:1905] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:41.439038 3813 ts_tablet_manager.cc:1905] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:41.441237 3814 ts_tablet_manager.cc:1918] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:41.441462 3814 log.cc:1196] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/0ead86afcbaf4e1d998b537a079ddbfd
I20260619 06:55:41.441952 3814 ts_tablet_manager.cc:1939] T 0ead86afcbaf4e1d998b537a079ddbfd P e1b3d875d5ac4ae69e6e565f5c12c125: Deleting consensus metadata
I20260619 06:55:41.442416 3813 ts_tablet_manager.cc:1918] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:41.442510 3813 log.cc:1196] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/0ead86afcbaf4e1d998b537a079ddbfd
I20260619 06:55:41.442847 3813 ts_tablet_manager.cc:1939] T 0ead86afcbaf4e1d998b537a079ddbfd P 6eb1b29dc1004f9fb5001ad252281d3e: Deleting consensus metadata
I20260619 06:55:41.444591 3274 catalog_manager.cc:4841] TS 6eb1b29dc1004f9fb5001ad252281d3e (127.22.113.3:40031): tablet 0ead86afcbaf4e1d998b537a079ddbfd (table test-table [id=a68a27c07597487f9856b58f1eb8a8b4]) successfully deleted
W20260619 06:55:41.447937 3275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:38305 (error 108)
I20260619 06:55:41.448217 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3490
I20260619 06:55:41.458490 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.458064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:38305 (local address 127.0.0.1:58924)
0619 06:55:41.458238 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:41.458355 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:55:41.458793 3275 catalog_manager.cc:4584] TS 7b7f999d926d43448a976623d9c11a1d (127.22.113.1:38305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ead86afcbaf4e1d998b537a079ddbfd: Network error: Client connection negotiation failed: client connection to 127.22.113.1:38305: connect: Connection refused (error 111)
I20260619 06:55:41.460433 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3623
I20260619 06:55:41.460512 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:41.460151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36409 (local address 127.0.0.1:54282)
0619 06:55:41.460328 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:41.460402 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:55:41.470407 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3255
I20260619 06:55:41.477844 22980 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:55:41.814097 22980 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:55:41.814240 22980 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:55:42.159991 22980 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:55:42.160 UTC [2322] LOG: received smart shutdown request
2026-06-19 06:55:42.164 UTC [2322] LOG: background worker "logical replication launcher" (PID 2331) exited with exit code 1
2026-06-19 06:55:42.164 UTC [2326] LOG: shutting down
2026-06-19 06:55:42.347 UTC [2322] LOG: database system is shut down
2026-06-19T06:55:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51971 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:55:42 dist-test-slave-g8zg krb5kdc[3824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:55:42 dist-test-slave-g8zg krb5kdc[3824](info): set up 2 sockets
Jun 19 06:55:42 dist-test-slave-g8zg krb5kdc[3824](info): commencing operation
krb5kdc: starting...
W20260619 06:55:43.535739 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:55:43 dist-test-slave-g8zg krb5kdc[3824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:55:43.670868 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39235
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:43.791927 3837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:43.792251 3837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:43.792369 3837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:43.796687 3837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:43.796777 3837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:43.796813 3837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:43.796867 3837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:43.796924 3837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:43.802417 3837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:43.803644 3837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:43.804718 3837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:43.814888 3845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:43.814868 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:43.814868 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:43.815424 3837 server_base.cc:1034] running on GCE node
I20260619 06:55:43.816248 3837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:43.817680 3837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:43.818889 3837 hybrid_clock.cc:648] HybridClock initialized: now 1781852143818854 us; error 64 us; skew 500 ppm
Jun 19 06:55:43 dist-test-slave-g8zg krb5kdc[3824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:43.822368 3837 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:43.823606 3837 webserver.cc:460] Webserver started at http://127.22.113.62:36399/ using document root <none> and password file <none>
I20260619 06:55:43.824301 3837 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:43.824397 3837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:43.824668 3837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:43.827088 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "6a408c16bcc441dcb725eee2f2ff7b42"
format_stamp: "Formatted at 2026-06-19 06:55:43 on dist-test-slave-g8zg"
server_key: "5928a70b25d83392f93802889763ccc9"
server_key_iv: "4f4748cc207721288c567d5865f207ea"
server_key_version: "encryptionkey@0"
I20260619 06:55:43.827772 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "6a408c16bcc441dcb725eee2f2ff7b42"
format_stamp: "Formatted at 2026-06-19 06:55:43 on dist-test-slave-g8zg"
server_key: "5928a70b25d83392f93802889763ccc9"
server_key_iv: "4f4748cc207721288c567d5865f207ea"
server_key_version: "encryptionkey@0"
I20260619 06:55:43.832211 3837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:55:43.835043 3852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:43.836772 3837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:55:43.836951 3837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "6a408c16bcc441dcb725eee2f2ff7b42"
format_stamp: "Formatted at 2026-06-19 06:55:43 on dist-test-slave-g8zg"
server_key: "5928a70b25d83392f93802889763ccc9"
server_key_iv: "4f4748cc207721288c567d5865f207ea"
server_key_version: "encryptionkey@0"
I20260619 06:55:43.837088 3837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:43.864601 3837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:43.868080 3837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:43.868309 3837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:43.876812 3837 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:39235
I20260619 06:55:43.876839 3904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:39235 every 8 connection(s)
I20260619 06:55:43.877903 3837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:43.879061 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3837
I20260619 06:55:43.879217 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:55:43.879582 22980 external_mini_cluster.cc:1428] Setting key 73028d210ff219b8d31228a2bd49e6e3
I20260619 06:55:43.881616 3905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:43.885691 3905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.61@KRBTEST.COM" created.
Jun 19 06:55:43 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:43.902904 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.888144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39235 (local address 127.0.0.1:54494)
0619 06:55:43.888832 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:43.888888 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:43.889969 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:43.890909 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:43.890915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:43.891383 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:43.891818 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:43.891848 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:43.893289 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:43.893292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:43.894091 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:43.894100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:43.894828 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:43.894832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:43.894944 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:43.895685 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:43.895703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:43.897378 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:43.899441 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:43.899450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:43.899461 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:43.899701 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:43.900041 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:43.900047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:43.900049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:43.900172 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:43.900878 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:43.900885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:43.901033 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 06:55:43.901225 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:55:43.901679 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":118,"spinlock_wait_cycles":56064,"thread_start_us":183,"threads_started":1}
I20260619 06:55:43.902904 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.888144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.22.113.62:39235)
0619 06:55:43.889198 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:43.889217 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:43.889259 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:55:43.890372 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:43.890377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:43.890584 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:43.890896 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:43.891996 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:43.893156 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:43.894229 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:43.894650 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:43.894834 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:43.897548 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:43.897554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:43.897566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:43.897593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:43.899322 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:43.899826 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:43.899832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:43.899837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:43.899944 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:43.900266 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:43.900272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:43.900274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:43.900755 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:43.900902 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:43.901285 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 06:55:43.901626 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":874,"thread_start_us":183,"threads_started":1}
I20260619 06:55:43.905915 3905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
I20260619 06:55:43.906982 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.906534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47610)
0619 06:55:43.906714 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:43.906826 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 06:55:43.907215 3854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
I20260619 06:55:43.907971 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:33275
--webserver_interface=127.22.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:43.908387 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.908093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47626)
0619 06:55:43.908274 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:43.908319 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260619 06:55:43.908558 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33275: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
I20260619 06:55:43.957851 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } attempt: 1
I20260619 06:55:43.958988 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.958567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47640)
0619 06:55:43.958788 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:43.958886 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:55:43.959905 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:43.959611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47656)
0619 06:55:43.959756 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:43.959826 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:55:43.960060 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33275: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
W20260619 06:55:44.022034 3912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:44.022341 3912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:44.022383 3912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:44.026595 3912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:44.026675 3912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:44.026695 3912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:44.026712 3912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:44.026727 3912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:44.028249 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } attempt: 2
I20260619 06:55:44.029264 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.028789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47668)
0619 06:55:44.029036 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.029140 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:55:44.030103 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.029906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47670)
0619 06:55:44.030006 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.030052 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:55:44.030411 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33275: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33275: connect: Connection refused (error 111)
I20260619 06:55:44.033022 3912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:33275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:44.034195 3912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:44.035310 3912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:44.044826 3918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:44.044842 3912 server_base.cc:1034] running on GCE node
W20260619 06:55:44.044854 3917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:44.045084 3920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:44.045782 3912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:44.047152 3912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:44.048323 3912 hybrid_clock.cc:648] HybridClock initialized: now 1781852144048290 us; error 53 us; skew 500 ppm
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:44.051698 3912 init.cc:377] Logged in from keytab as kudu/127.22.113.61@KRBTEST.COM (short username kudu)
I20260619 06:55:44.052743 3912 webserver.cc:460] Webserver started at http://127.22.113.61:35465/ using document root <none> and password file <none>
I20260619 06:55:44.053383 3912 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:44.053447 3912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:44.053664 3912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:44.056161 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data/instance:
uuid: "3809b45ef2ea42f795c7b4d8228d3e35"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "61d28cbd01e1a705bd66d00f9de88ba3"
server_key_iv: "27c3d735b1c8ce7a054cf1ecf30a9e23"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.056747 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal/instance:
uuid: "3809b45ef2ea42f795c7b4d8228d3e35"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "61d28cbd01e1a705bd66d00f9de88ba3"
server_key_iv: "27c3d735b1c8ce7a054cf1ecf30a9e23"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.061184 3912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:55:44.064131 3927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:44.066197 3912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:55:44.066370 3912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "3809b45ef2ea42f795c7b4d8228d3e35"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "61d28cbd01e1a705bd66d00f9de88ba3"
server_key_iv: "27c3d735b1c8ce7a054cf1ecf30a9e23"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.066522 3912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:44.078675 3912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:44.081784 3912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:44.082003 3912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:44.089737 3912 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.61:33275
I20260619 06:55:44.089782 3979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.61:33275 every 8 connection(s)
I20260619 06:55:44.090761 3912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
I20260619 06:55:44.094432 3980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:44.095522 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3912
I20260619 06:55:44.095681 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-1/wal/instance
I20260619 06:55:44.096050 22980 external_mini_cluster.cc:1428] Setting key 4bf8a6972bcb8d2f974cfa25b7c2a189
I20260619 06:55:44.100126 3980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.60@KRBTEST.COM" created.
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:44.117365 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.103346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.22.113.62:39235)
0619 06:55:44.103530 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.103534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.104790 (+ 1256us) server_negotiation.cc:407] Connection header received
0619 06:55:44.105940 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.105943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.106012 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.106120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.107603 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.108177 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.109421 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.109677 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.109792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.112604 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.112607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.112610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.112639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.114363 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.114897 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.114906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.114907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.114983 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.115386 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.115389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.115391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.115596 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.115714 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.117027 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.117183 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:55:44.117877 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.103915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39235 (local address 127.0.0.1:54508)
0619 06:55:44.104590 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.104648 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.105702 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.106329 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.106337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.106755 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.107449 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.107467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.108311 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.108319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.109289 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.109299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.109843 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.109848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.110029 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.110722 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.110740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.112416 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.114486 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.114492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.114510 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.114785 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.115083 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.115086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.115088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.115295 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.115711 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.115718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.116026 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.116237 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.116670 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":186,"threads_started":1}
I20260619 06:55:44.119665 3980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:55:44.121628 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
I20260619 06:55:44.124877 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:39017
--webserver_interface=127.22.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:55:44.134032 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.120426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47678)
0619 06:55:44.120574 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.120595 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.120683 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.121520 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.121526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.121723 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.121852 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.121859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.123110 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.123122 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.123795 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.123806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.124588 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.124592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.124737 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.125322 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.125341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.127113 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.132245 (+ 5132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.132251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.132532 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.132798 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.132803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.132883 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.133399 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.133402 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.133502 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.133640 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.133811 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":15,"spinlock_wait_cycles":8576}
I20260619 06:55:44.134043 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.120494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.22.113.61:33275)
0619 06:55:44.120995 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.121006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.121030 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:55:44.121087 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.121092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.121293 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.121508 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.121977 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.122993 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.123918 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.124467 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.124600 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.127280 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.127291 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.127310 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.127335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.132126 (+ 4791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132609 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.132613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.132615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.132700 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132969 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.132975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.132978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.133283 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.133441 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.133747 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.133895 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260619 06:55:44.135046 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.122212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.22.113.61:33275)
0619 06:55:44.122630 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.122637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.122648 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:55:44.122678 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.122679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.122714 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.122812 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.123358 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.124364 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.125665 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.125909 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.126031 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.129073 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.129083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.129089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.129126 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.131440 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132228 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.132239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.132240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.132407 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132888 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.132894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.132899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.133344 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.133467 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.134578 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.134742 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"spinlock_wait_cycles":11392,"thread_start_us":186,"threads_started":1}
I20260619 06:55:44.135305 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.122155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47686)
0619 06:55:44.122268 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.122283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.122371 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.122829 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.122832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.123063 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.123194 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.123200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.124492 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.124497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.125538 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.125548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.126033 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.126036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.126143 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.126606 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.126629 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.128335 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.131645 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.131651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.131656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.132039 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.132573 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.132579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.132583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.132735 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.133543 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.133549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.133691 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.133818 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.134044 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 06:55:44.136664 3905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:55:44.136898 3980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:55:44.137606 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.137321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52622)
0619 06:55:44.137471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.137531 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:55:44.138139 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.137848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52632)
0619 06:55:44.137972 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.138045 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 06:55:44.138531 3929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
I20260619 06:55:44.138751 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.138464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52638)
0619 06:55:44.138649 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.138700 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 06:55:44.138906 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
I20260619 06:55:44.139592 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.139396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52648)
0619 06:55:44.139499 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.139543 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:55:44.139880 3980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
I20260619 06:55:44.170069 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 1
I20260619 06:55:44.171317 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.170807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52650)
0619 06:55:44.171057 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.171142 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 06:55:44.172116 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.171904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52654)
0619 06:55:44.172003 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.172053 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:55:44.172255 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
I20260619 06:55:44.189160 3980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 1
I20260619 06:55:44.190272 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.189898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52668)
0619 06:55:44.190095 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.190177 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:55:44.191289 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.190931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52670)
0619 06:55:44.191138 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.191193 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260619 06:55:44.191426 3980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
W20260619 06:55:44.243505 3987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:44.243860 3987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:44.243973 3987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:55:44.247444 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 2
W20260619 06:55:44.248353 3987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:44.248458 3987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:44.248504 3987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:44.248540 3987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:44.248574 3987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:44.248695 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.248165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52674)
0619 06:55:44.248386 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.248552 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 06:55:44.249543 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.249317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52686)
0619 06:55:44.249422 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.249479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:55:44.249667 3905 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
I20260619 06:55:44.254367 3987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:39235,127.22.113.61:33275,127.22.113.60:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:44.255594 3987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:44.256711 3987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:55:44.259626 3980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 2
I20260619 06:55:44.260594 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.260254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52696)
0619 06:55:44.260403 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.260474 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:55:44.261565 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.261298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52702)
0619 06:55:44.261442 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.261500 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:55:44.261735 3980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:39017: Network error: Client connection negotiation failed: client connection to 127.22.113.60:39017: connect: Connection refused (error 111)
W20260619 06:55:44.265874 3994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:44.265974 3993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:44.265873 3996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:44.266423 3987 server_base.cc:1034] running on GCE node
I20260619 06:55:44.267021 3987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:44.268231 3987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:44.269407 3987 hybrid_clock.cc:648] HybridClock initialized: now 1781852144269367 us; error 58 us; skew 500 ppm
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:44.273103 3987 init.cc:377] Logged in from keytab as kudu/127.22.113.60@KRBTEST.COM (short username kudu)
I20260619 06:55:44.274175 3987 webserver.cc:460] Webserver started at http://127.22.113.60:45073/ using document root <none> and password file <none>
I20260619 06:55:44.274773 3987 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:44.274866 3987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:44.275115 3987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:44.277251 3987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data/instance:
uuid: "3862a725e70746628ad9c641cbdc5f52"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "1f962cbe7b9efe8c2f169ef8aa6960aa"
server_key_iv: "d749c248b14f3e2d4d44fed22772e20c"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.277837 3987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal/instance:
uuid: "3862a725e70746628ad9c641cbdc5f52"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "1f962cbe7b9efe8c2f169ef8aa6960aa"
server_key_iv: "d749c248b14f3e2d4d44fed22772e20c"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.281924 3987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:55:44.284894 4003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:44.286466 3987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:55:44.286641 3987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "3862a725e70746628ad9c641cbdc5f52"
format_stamp: "Formatted at 2026-06-19 06:55:44 on dist-test-slave-g8zg"
server_key: "1f962cbe7b9efe8c2f169ef8aa6960aa"
server_key_iv: "d749c248b14f3e2d4d44fed22772e20c"
server_key_version: "encryptionkey@0"
I20260619 06:55:44.286816 3987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:44.302083 3987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:44.305691 3987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:44.305922 3987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:44.313992 3987 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.60:39017
I20260619 06:55:44.314062 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.60:39017 every 8 connection(s)
I20260619 06:55:44.314996 3987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
I20260619 06:55:44.318542 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:44.323424 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:55:44.324352 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 3987
I20260619 06:55:44.324471 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851976522130-22980-0/minicluster-data/master-2/wal/instance
I20260619 06:55:44.324926 22980 external_mini_cluster.cc:1428] Setting key 35bc069451b4d4a6053cb4d280434a80
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:55:44.338613 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.326073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.22.113.62:39235)
0619 06:55:44.326216 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.326221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.327180 (+ 959us) server_negotiation.cc:407] Connection header received
0619 06:55:44.328147 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.328150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.328215 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.328319 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.329436 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.330000 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.331177 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.331487 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.331587 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.334552 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.334555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.334558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.334581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.336284 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.336936 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.336941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.336943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.337025 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.337391 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.337394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.337395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.337599 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.337706 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.338312 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.338441 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:55:44.339790 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.326423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39235 (local address 127.0.0.1:54512)
0619 06:55:44.327009 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.327052 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.327920 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.328410 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.328416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.328912 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.329271 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.329284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.330116 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.330120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.330990 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.330998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.331655 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.331659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.331770 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.332560 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.332576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.334394 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.336433 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.336447 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.336461 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.336830 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.337125 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.337128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.337130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.337287 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.337721 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.337730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.338073 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.338255 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.338616 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":54,"thread_start_us":159,"threads_started":1}
I20260619 06:55:44.339797 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.326422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.22.113.60:39017)
0619 06:55:44.326954 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.326975 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.327021 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:55:44.327902 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.327923 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.328154 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.328353 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.329160 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.330236 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.331081 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.331566 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.331732 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.334898 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.334904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.334916 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.334937 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.336758 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.337275 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.337280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.337281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.337363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.337654 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.337658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.337660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.338083 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.338314 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.338558 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.338728 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":393,"spinlock_wait_cycles":43008,"thread_start_us":202,"threads_started":1}
I20260619 06:55:44.341966 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:44.348822 3905 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 3
I20260619 06:55:44.352908 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.342757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.22.113.61:33275)
0619 06:55:44.342903 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.342908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.342928 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:55:44.342984 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.342986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.343076 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.343169 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.343742 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.344536 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.345522 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.345913 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.346025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.348520 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.348524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.348527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.348554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.351026 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.351578 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.351584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.351586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.351674 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.351967 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.351970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.351972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.352189 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.352299 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.352589 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.352722 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:55:44.353044 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.342574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33275 (local address 127.0.0.1:47692)
0619 06:55:44.342693 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.342711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.342866 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.343225 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.343228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.343461 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.343615 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.343621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.344750 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.344753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.345404 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.345413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.346057 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.346061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.346171 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.346828 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.346840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.348364 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.351155 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.351160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.351163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.351469 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.351773 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.351777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.351779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.351865 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.352287 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.352290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.352400 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.352523 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.352701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:55:44.352883 3980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } attempt: 3
I20260619 06:55:44.354576 4056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:55:44.355005 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.342883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.22.113.62:39235)
0619 06:55:44.342994 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.342998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.343116 (+ 118us) server_negotiation.cc:407] Connection header received
0619 06:55:44.343267 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.343273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.343343 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.343437 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.344252 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.344958 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.345951 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.346304 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.346431 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.349452 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.349456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.349458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.349479 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.351699 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.352288 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.352292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.352294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.352470 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.352830 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.352833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.352835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.353084 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.353309 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.354233 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.354366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":49}
I20260619 06:55:44.359525 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.349322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52720)
0619 06:55:44.349556 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.349570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.349647 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.349907 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.349910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.350085 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.350160 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.350165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.351208 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.351212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.352050 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.352060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.352830 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.352833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.352960 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.353879 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.353893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.355330 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.357892 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.357895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.357897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.358149 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.358468 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.358472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.358474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.358557 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.358987 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.358990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.359090 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.359170 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.359352 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":5}
I20260619 06:55:44.360190 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.349449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.22.113.60:39017)
0619 06:55:44.349603 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.349610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.349627 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:55:44.349725 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.349730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.349781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.349886 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.350274 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.351059 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.352200 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.352694 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.353164 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.355485 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.355488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.355490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.355516 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.357765 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.358266 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.358271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.358273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.358356 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.358666 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.358669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.358671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.358866 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.359843 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.359913 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.360021 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852144, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:55:44.364662 3905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: Bootstrap starting.
Jun 19 06:55:44 dist-test-slave-g8zg krb5kdc[3824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:55:44.366506 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.353704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52736)
0619 06:55:44.353927 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.353944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.354037 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.354757 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.354760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.354970 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.355091 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.355098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.356747 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.356753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.357882 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.357891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.359874 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.359878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.360246 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.360812 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.360832 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.362435 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.364568 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.364572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.364574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.364788 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.365180 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.365185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.365188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.365286 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.365890 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.365893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.366008 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.366101 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.366277 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":265}
I20260619 06:55:44.367269 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.353743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.22.113.60:39017)
0619 06:55:44.354525 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.354535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.354547 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:55:44.354577 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.354579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.354624 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.354743 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.355397 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.356637 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.358012 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.358383 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.360017 (+ 1634us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.362593 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.362597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.362599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.362623 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.364448 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.364913 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.364917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.364919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.365067 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.365410 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.365413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.365415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.365752 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.366674 (+ 922us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.366794 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.367031 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"spinlock_wait_cycles":1920,"thread_start_us":131,"threads_started":1}
I20260619 06:55:44.368870 3905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:44.369300 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.355139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:39017 (local address 127.0.0.1:52750)
0619 06:55:44.356818 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:44.356831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:44.356900 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:44.357908 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:44.357911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:44.358080 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:44.358191 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.358201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.359293 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.359296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.360059 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:44.360068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.361348 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.361355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:44.361460 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.361893 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:44.361906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:44.363406 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:44.367232 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.367235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.367237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.367818 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.368103 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:44.368106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:44.368108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:44.368190 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:44.368659 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:44.368662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:44.368787 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 06:55:44.368904 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:55:44.369061 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1617}
I20260619 06:55:44.369340 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.355531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.22.113.60:39017)
0619 06:55:44.356192 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.356202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.356977 (+ 775us) server_negotiation.cc:407] Connection header received
0619 06:55:44.357019 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.357023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.357068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.357159 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.358326 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.359170 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.360175 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.361202 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.361306 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.364895 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.364899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.364902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.364925 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.367117 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.367924 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.367927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.367929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.368012 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.368316 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.368319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.368321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.368528 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.368637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.369003 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.369125 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":111,"threads_started":1}
I20260619 06:55:44.370057 3905 log.cc:824] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:44.371110 3980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: Bootstrap starting.
I20260619 06:55:44.372774 3905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: No bootstrap required, opened a new log
I20260619 06:55:44.372794 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: Bootstrap starting.
I20260619 06:55:44.373564 3980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:44.374274 3980 log.cc:824] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:44.375576 4056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:44.376567 4056 log.cc:824] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:44.376889 3980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: No bootstrap required, opened a new log
I20260619 06:55:44.377480 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.356269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.22.113.61:33275)
0619 06:55:44.356430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.356436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.356452 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:55:44.356572 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.356575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.356628 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.356711 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.357591 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.358314 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.359582 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.359895 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.359994 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.366226 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.366232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.366238 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.366263 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.375158 (+ 8895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.375763 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.375767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.375768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.375866 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.376162 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.376165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.376167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.376381 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.376538 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.377196 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.377311 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":378,"server-negotiator.queue_time_us":71}
I20260619 06:55:44.377472 3905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.377777 3905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:44.377840 3905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a408c16bcc441dcb725eee2f2ff7b42, State: Initialized, Role: FOLLOWER
I20260619 06:55:44.378779 3905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.379469 4066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.379706 4066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.380303 3905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:44.381625 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:55:44.381731 3980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.381969 3980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:44.382043 3980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3809b45ef2ea42f795c7b4d8228d3e35, State: Initialized, Role: FOLLOWER
I20260619 06:55:44.382639 3980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.385296 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: No bootstrap required, opened a new log
I20260619 06:55:44.385280 3980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:55:44.385895 4077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:55:44.386021 4077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:55:44.386377 4083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.386550 4083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.385670 3837 master_runner.cc:186] Error getting master registration for 127.22.113.61:33275: OK, instance_id { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" instance_seqno: 1781852144087549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:55:44.389642 3837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:55:44.390667 4056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.390976 4056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:44.391070 4056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3862a725e70746628ad9c641cbdc5f52, State: Initialized, Role: FOLLOWER
I20260619 06:55:44.391649 4056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.393168 4056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:55:44.394609 4094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:55:44.394740 4094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:55:44.401240 3987 master_runner.cc:263] No leader master found from master 3862a725e70746628ad9c641cbdc5f52
I20260619 06:55:44.401343 3987 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:55:44.393033 3912 master_runner.cc:186] Error getting master registration for 127.22.113.60:39017: OK, instance_id { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" instance_seqno: 1781852144311605 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:55:44.402643 3912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:55:44.402824 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.403111 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 06:55:44.408272 4107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:55:44.410175 4107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:55:44.413077 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.396508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.22.113.61:33275)
0619 06:55:44.396635 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.396641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.397806 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 06:55:44.398023 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.398026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.398078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.398133 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.399267 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.400097 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.401341 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.401676 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.401754 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.404990 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.404993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.405008 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.405035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.407564 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.408364 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.408370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.408374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.408476 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.410489 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.410492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.410494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.410731 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.410837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.412758 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.412904 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:55:44.414556 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.397018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.22.113.60:39017)
0619 06:55:44.397136 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.397141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.403114 (+ 5973us) server_negotiation.cc:407] Connection header received
0619 06:55:44.403366 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.403369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.403419 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.403474 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.404415 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.405226 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.406403 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.406769 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.406841 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.409521 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.409525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.409528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.409553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.411610 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.412339 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.412342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.412344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.412425 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.413582 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.413585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.413587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.413792 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.413915 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.414260 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.414405 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:55:44.416497 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:44.396013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.22.113.62:39235)
0619 06:55:44.396163 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:44.396168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:44.402534 (+ 6366us) server_negotiation.cc:407] Connection header received
0619 06:55:44.404023 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:44.404027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:44.404088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:44.404142 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:44.405402 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.406185 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.408604 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:44.408936 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:44.409028 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:44.410790 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:44.410794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:44.410797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:44.410828 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:44.412535 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.413053 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.413060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.413062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.413147 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:44.415374 (+ 2227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:44.415378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:44.415380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:44.415645 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:44.415758 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:44.416156 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 06:55:44.416299 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:55:44.551681 4095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:44.551959 4095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.553017 4095 leader_election.cc:288] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a408c16bcc441dcb725eee2f2ff7b42 (127.22.113.62:39235), 3809b45ef2ea42f795c7b4d8228d3e35 (127.22.113.61:33275)
I20260619 06:55:44.553589 3879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3862a725e70746628ad9c641cbdc5f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" is_pre_election: true
I20260619 06:55:44.553730 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3862a725e70746628ad9c641cbdc5f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" is_pre_election: true
I20260619 06:55:44.553906 3879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3862a725e70746628ad9c641cbdc5f52 in term 0.
I20260619 06:55:44.553995 3953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3862a725e70746628ad9c641cbdc5f52 in term 0.
I20260619 06:55:44.554378 4004 leader_election.cc:302] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3809b45ef2ea42f795c7b4d8228d3e35, 3862a725e70746628ad9c641cbdc5f52, 6a408c16bcc441dcb725eee2f2ff7b42; no voters:
I20260619 06:55:44.554661 4095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:55:44.554739 4095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:55:44.554781 4095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:44.556087 4095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.556440 4095 leader_election.cc:288] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [CANDIDATE]: Term 1 election: Requested vote from peers 6a408c16bcc441dcb725eee2f2ff7b42 (127.22.113.62:39235), 3809b45ef2ea42f795c7b4d8228d3e35 (127.22.113.61:33275)
I20260619 06:55:44.556778 3879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3862a725e70746628ad9c641cbdc5f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a408c16bcc441dcb725eee2f2ff7b42"
I20260619 06:55:44.556896 3879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:44.556926 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3862a725e70746628ad9c641cbdc5f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3809b45ef2ea42f795c7b4d8228d3e35"
I20260619 06:55:44.557031 3953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 0 FOLLOWER]: Advancing to term 1
W20260619 06:55:44.557613 4009 tablet.cc:2369] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:55:44.558220 3953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3862a725e70746628ad9c641cbdc5f52 in term 1.
I20260619 06:55:44.558220 3879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3862a725e70746628ad9c641cbdc5f52 in term 1.
I20260619 06:55:44.558557 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3809b45ef2ea42f795c7b4d8228d3e35, 3862a725e70746628ad9c641cbdc5f52; no voters:
I20260619 06:55:44.558786 4095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:44.559037 4095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [term 1 LEADER]: Becoming Leader. State: Replica: 3862a725e70746628ad9c641cbdc5f52, State: Running, Role: LEADER
I20260619 06:55:44.559412 4095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } }
I20260619 06:55:44.561724 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3862a725e70746628ad9c641cbdc5f52. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.561875 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:44.562250 4114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:44.563354 4114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:44.567665 3953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [term 1 FOLLOWER]: Refusing update from remote peer 3862a725e70746628ad9c641cbdc5f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:44.567700 3879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [term 1 FOLLOWER]: Refusing update from remote peer 3862a725e70746628ad9c641cbdc5f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:44.568329 4112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:44.568639 4095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:44.572149 4066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3862a725e70746628ad9c641cbdc5f52. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.572273 4066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.572399 4083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3862a725e70746628ad9c641cbdc5f52. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.572505 4083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.576177 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.576314 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:44.576337 4066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.576432 4066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.576419 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.576490 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:44.577793 4083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3862a725e70746628ad9c641cbdc5f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a408c16bcc441dcb725eee2f2ff7b42" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39235 } } peers { permanent_uuid: "3809b45ef2ea42f795c7b4d8228d3e35" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33275 } } peers { permanent_uuid: "3862a725e70746628ad9c641cbdc5f52" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 39017 } } }
I20260619 06:55:44.577913 4083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:55:44.577920 4116 mvcc.cc:205] Tried to move back new op lower bound from 7298466384143187968 to 7298466384118992896. Current Snapshot: MvccSnapshot[applied={T|T < 7298466384143187968}]
I20260619 06:55:44.579561 4115 mvcc.cc:205] Tried to move back new op lower bound from 7298466384143187968 to 7298466384118992896. Current Snapshot: MvccSnapshot[applied={T|T < 7298466384143187968}]
I20260619 06:55:44.580430 4117 mvcc.cc:205] Tried to move back new op lower bound from 7298466384143187968 to 7298466384118992896. Current Snapshot: MvccSnapshot[applied={T|T < 7298466384143187968}]
I20260619 06:55:44.581880 4114 catalog_manager.cc:1347] Generated new cluster ID: df08d2348a2743db89e1d0197a11dc96
I20260619 06:55:44.581939 4114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:44.602684 4114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:55:44.603729 4114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:44.614959 4114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52: Generated new TSK 0
I20260619 06:55:44.615772 4114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:55:45.387660 4077 catalog_manager.cc:1259] Loaded cluster ID: df08d2348a2743db89e1d0197a11dc96
I20260619 06:55:45.387748 4077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: loading cluster ID for follower catalog manager: success
I20260619 06:55:45.389369 4077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: acquiring CA information for follower catalog manager: success
I20260619 06:55:45.390610 4077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a408c16bcc441dcb725eee2f2ff7b42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:55:45.396476 4094 catalog_manager.cc:1259] Loaded cluster ID: df08d2348a2743db89e1d0197a11dc96
I20260619 06:55:45.396572 4094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: loading cluster ID for follower catalog manager: success
I20260619 06:55:45.398393 4094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: acquiring CA information for follower catalog manager: success
I20260619 06:55:45.399744 4094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3809b45ef2ea42f795c7b4d8228d3e35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:55:45.674839 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3837
I20260619 06:55:45.681538 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3912
I20260619 06:55:45.682566 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:45.681718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39235 (local address 127.0.0.1:54546)
0619 06:55:45.682377 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:45.682472 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:39235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":89,"threads_started":1}
W20260619 06:55:45.682832 4004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.62:39235: connect: Connection refused (error 111)
I20260619 06:55:45.683766 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:45.683556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:39235 (local address 127.0.0.1:54552)
0619 06:55:45.683667 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:45.683719 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:39235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:55:45.683957 4004 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3862a725e70746628ad9c641cbdc5f52 -> Peer 6a408c16bcc441dcb725eee2f2ff7b42 (127.22.113.62:39235): Couldn't send request to peer 6a408c16bcc441dcb725eee2f2ff7b42. Status: Network error: Client connection negotiation failed: client connection to 127.22.113.62:39235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:55:45.689168 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 3987
2026-06-19T06:55:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3216 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:55:45 dist-test-slave-g8zg krb5kdc[4131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:55:45 dist-test-slave-g8zg krb5kdc[4131](info): set up 2 sockets
Jun 19 06:55:45 dist-test-slave-g8zg krb5kdc[4131](info): commencing operation
krb5kdc: starting...
W20260619 06:55:46.767668 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:55:46 dist-test-slave-g8zg krb5kdc[4131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:55:46.906527 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40841
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:47.020238 4144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:47.020555 4144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:47.020597 4144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:47.024636 4144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:47.024708 4144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:47.024727 4144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:47.024744 4144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:47.024760 4144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:47.030149 4144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:47.031237 4144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:47.032366 4144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:47.041707 4152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:47.041858 4144 server_base.cc:1034] running on GCE node
W20260619 06:55:47.041775 4150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:47.041772 4149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:47.042935 4144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:47.044605 4144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:47.045800 4144 hybrid_clock.cc:648] HybridClock initialized: now 1781852147045765 us; error 54 us; skew 500 ppm
Jun 19 06:55:47 dist-test-slave-g8zg krb5kdc[4131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852147, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:47.049202 4144 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:47.050333 4144 webserver.cc:460] Webserver started at http://127.22.113.62:37751/ using document root <none> and password file <none>
I20260619 06:55:47.051023 4144 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:47.051088 4144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:47.051424 4144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:47.053931 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "8dffb6dd21414145857c1dea435795c7"
format_stamp: "Formatted at 2026-06-19 06:55:47 on dist-test-slave-g8zg"
server_key: "dc9170da8467a702c103340e1511e2e7"
server_key_iv: "b573a4e8f4c038488a5ef6f9ffd11c68"
server_key_version: "encryptionkey@0"
I20260619 06:55:47.054570 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "8dffb6dd21414145857c1dea435795c7"
format_stamp: "Formatted at 2026-06-19 06:55:47 on dist-test-slave-g8zg"
server_key: "dc9170da8467a702c103340e1511e2e7"
server_key_iv: "b573a4e8f4c038488a5ef6f9ffd11c68"
server_key_version: "encryptionkey@0"
I20260619 06:55:47.059105 4144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 06:55:47.062203 4159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:47.064018 4144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:47.064177 4144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "8dffb6dd21414145857c1dea435795c7"
format_stamp: "Formatted at 2026-06-19 06:55:47 on dist-test-slave-g8zg"
server_key: "dc9170da8467a702c103340e1511e2e7"
server_key_iv: "b573a4e8f4c038488a5ef6f9ffd11c68"
server_key_version: "encryptionkey@0"
I20260619 06:55:47.064319 4144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:47.091670 4144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:47.095086 4144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:47.095345 4144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:47.103816 4144 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40841
I20260619 06:55:47.103997 4211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40841 every 8 connection(s)
I20260619 06:55:47.105046 4144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:47.108265 4212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:47.114213 4212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7: Bootstrap starting.
I20260619 06:55:47.114838 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4144
I20260619 06:55:47.115020 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:55:47.115434 22980 external_mini_cluster.cc:1428] Setting key f6bb5af0ae4d8d28eb291e243f3bc8cd
I20260619 06:55:47.117223 4212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:47.118022 4212 log.cc:824] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:47.120673 4212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7: No bootstrap required, opened a new log
I20260619 06:55:47.125459 4212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } }
I20260619 06:55:47.125782 4212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:47.125874 4212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dffb6dd21414145857c1dea435795c7, State: Initialized, Role: FOLLOWER
Jun 19 06:55:47 dist-test-slave-g8zg krb5kdc[4131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:47.126403 4212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } }
I20260619 06:55:47.126569 4212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:47.126652 4212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:47.126784 4212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:47.127916 4212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } }
I20260619 06:55:47.128357 4212 leader_election.cc:302] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dffb6dd21414145857c1dea435795c7; no voters:
I20260619 06:55:47.128778 4212 leader_election.cc:288] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:55:47.128868 4217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:47.129081 4217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8dffb6dd21414145857c1dea435795c7, State: Running, Role: LEADER
I20260619 06:55:47.129516 4217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } }
I20260619 06:55:47.130220 4212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:47.131525 4218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dffb6dd21414145857c1dea435795c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } } }
I20260619 06:55:47.131716 4218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:47.132061 4219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dffb6dd21414145857c1dea435795c7. Latest consensus state: current_term: 1 leader_uuid: "8dffb6dd21414145857c1dea435795c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dffb6dd21414145857c1dea435795c7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40841 } } }
I20260619 06:55:47.132181 4219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:47.132220 4226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:47.135557 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:47.116839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.22.113.62:40841)
0619 06:55:47.117403 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:47.117433 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:47.117478 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:55:47.118367 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:47.118391 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:47.118899 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:47.119360 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:47.120352 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:47.121521 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:47.122761 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:47.123412 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:47.123576 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:47.127075 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:47.127081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:47.127099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:47.127134 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:47.129584 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:47.130176 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:47.130183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:47.130191 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:47.130416 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:47.132337 (+ 1921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:47.132340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:47.132342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:47.132985 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:47.133892 (+ 907us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:47.134115 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:55:47.134515 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":182,"threads_started":1}
I20260619 06:55:47.136085 4226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:47.139106 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:55:47.143395 4226 catalog_manager.cc:1347] Generated new cluster ID: 9700e93f35544aecafc8d4728937c380
I20260619 06:55:47.143464 4226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:47.154526 4226 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:55:47.156093 4226 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:47.161995 4226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dffb6dd21414145857c1dea435795c7: Generated new TSK 0
I20260619 06:55:47.163028 4226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:55:47.196385 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4144
2026-06-19T06:55:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1506 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:55:47 dist-test-slave-g8zg krb5kdc[4239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:55:47 dist-test-slave-g8zg krb5kdc[4239](info): set up 2 sockets
Jun 19 06:55:47 dist-test-slave-g8zg krb5kdc[4239](info): commencing operation
krb5kdc: starting...
W20260619 06:55:48.259625 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:55:48 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:55:48.398886 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:48.530393 4252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:48.530726 4252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:48.530804 4252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:48.534994 4252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:48.535102 4252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:48.535138 4252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:48.535179 4252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:48.535216 4252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:48.540701 4252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:48.541851 4252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:48.542894 4252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:48.551717 4260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:48.551717 4257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:48.551780 4258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:48.552318 4252 server_base.cc:1034] running on GCE node
I20260619 06:55:48.552927 4252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:48.554113 4252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:48.555328 4252 hybrid_clock.cc:648] HybridClock initialized: now 1781852148555286 us; error 53 us; skew 500 ppm
Jun 19 06:55:48 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852148, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:48.558341 4252 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:48.559371 4252 webserver.cc:460] Webserver started at http://127.22.113.62:35207/ using document root <none> and password file <none>
I20260619 06:55:48.559974 4252 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:48.560066 4252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:48.560304 4252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:48.562366 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "094abe28bdc24678b8c74b074a1b01cd"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "b37773f704ae5b854858288d5a7bb0c2"
server_key_iv: "3cb1597b74715e95646e1d24b45e07b8"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.562932 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "094abe28bdc24678b8c74b074a1b01cd"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "b37773f704ae5b854858288d5a7bb0c2"
server_key_iv: "3cb1597b74715e95646e1d24b45e07b8"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.566888 4252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:55:48.569622 4267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:48.571103 4252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:55:48.571327 4252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "094abe28bdc24678b8c74b074a1b01cd"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "b37773f704ae5b854858288d5a7bb0c2"
server_key_iv: "3cb1597b74715e95646e1d24b45e07b8"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.571471 4252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:48.600210 4252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:48.603339 4252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:48.603564 4252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:48.611322 4252 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:38627
I20260619 06:55:48.611676 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:38627 every 8 connection(s)
I20260619 06:55:48.612388 4252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:48.615855 4320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:48.616822 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4252
I20260619 06:55:48.616976 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:55:48.617372 22980 external_mini_cluster.cc:1428] Setting key 995d59dd2e8471af627202a770519ae8
I20260619 06:55:48.623751 4320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap starting.
I20260619 06:55:48.627000 4320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Neither blocks nor log segments found. Creating new log.
Jun 19 06:55:48 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:48.627990 4320 log.cc:824] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:48.630311 4320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: No bootstrap required, opened a new log
I20260619 06:55:48.633980 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:48.619127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.22.113.62:38627)
0619 06:55:48.619635 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:48.619651 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:48.619711 (+ 60us) server_negotiation.cc:407] Connection header received
0619 06:55:48.620534 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:48.620551 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:48.620906 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:48.621212 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:48.622200 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.623295 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.624525 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.625057 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.625235 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:48.628602 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:48.628608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:48.628623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:48.628655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:48.630601 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:48.631209 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:48.631215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:48.631226 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:48.631401 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:48.631715 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:48.631718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:48.631719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:48.632182 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:48.632340 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:48.632690 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 06:55:48.633030 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":139,"threads_started":1}
I20260619 06:55:48.634891 4320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:48.635175 4320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:48.635251 4320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Initialized, Role: FOLLOWER
I20260619 06:55:48.635752 4320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:48.635923 4320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:48.636004 4320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:48.636132 4320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:48.637172 4320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:48.637531 4320 leader_election.cc:302] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094abe28bdc24678b8c74b074a1b01cd; no voters:
I20260619 06:55:48.637910 4320 leader_election.cc:288] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:55:48.638067 4325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:48.638347 4325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 LEADER]: Becoming Leader. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Running, Role: LEADER
I20260619 06:55:48.638808 4325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:48.639322 4320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:48.640913 4326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:48.641060 4326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:48.641424 4327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094abe28bdc24678b8c74b074a1b01cd. Latest consensus state: current_term: 1 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:48.641505 4327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
W20260619 06:55:48.645215 4340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:55:48.645320 4340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:55:48.645391 4335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:48.646198 4335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:48.652750 4335 catalog_manager.cc:1347] Generated new cluster ID: 96ba8d67202e48f0a4664b996a6e1894
I20260619 06:55:48.652833 4335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:48.663918 4335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:55:48.665040 4335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:48.669516 4335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Generated new TSK 0
I20260619 06:55:48.670295 4335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:55:48.708374 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:48.859799 4346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:48.860076 4346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:48.860121 4346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:48.864174 4346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:48.864244 4346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:48.864349 4346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:55:48.869592 4346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:48.870759 4346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:48.871838 4346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:48.882241 4352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:48.882354 4351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:48.882277 4354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:48.882546 4346 server_base.cc:1034] running on GCE node
I20260619 06:55:48.883291 4346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:48.884161 4346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:48.885360 4346 hybrid_clock.cc:648] HybridClock initialized: now 1781852148885340 us; error 37 us; skew 500 ppm
Jun 19 06:55:48 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852148, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:48.888759 4346 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:55:48.890053 4346 webserver.cc:460] Webserver started at http://127.22.113.1:32857/ using document root <none> and password file <none>
I20260619 06:55:48.890938 4346 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:48.891016 4346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:48.891335 4346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:48.893880 4346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "229e3e6391d74f73a9eabeaddc832bb7"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "765762330a28a59aff1abf1854799a46"
server_key_iv: "779d041c57b0ea40070a552297bc1adb"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.894534 4346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "229e3e6391d74f73a9eabeaddc832bb7"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "765762330a28a59aff1abf1854799a46"
server_key_iv: "779d041c57b0ea40070a552297bc1adb"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.899386 4346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:55:48.902362 4361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:48.904058 4346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:48.904203 4346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "229e3e6391d74f73a9eabeaddc832bb7"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "765762330a28a59aff1abf1854799a46"
server_key_iv: "779d041c57b0ea40070a552297bc1adb"
server_key_version: "encryptionkey@0"
I20260619 06:55:48.904330 4346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:48.931147 4346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:48.935016 4346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:48.935269 4346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:48.936053 4346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:48.937047 4346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:48.937120 4346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:48.937191 4346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:48.937235 4346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:48.949471 4346 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:34965
I20260619 06:55:48.949532 4474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:34965 every 8 connection(s)
I20260619 06:55:48.950584 4346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:55:48.958278 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4346
I20260619 06:55:48.958455 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:55:48.958794 22980 external_mini_cluster.cc:1428] Setting key 5c7d481920028fb0d53095327e53b06c
Jun 19 06:55:48 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852148, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:48.966715 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:48.952868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44063 (local address 127.22.113.62:38627)
0619 06:55:48.953022 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:48.953026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:48.954143 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 06:55:48.955683 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:48.955686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:48.955755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:48.955857 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:48.957210 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.957801 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.959176 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.959484 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.959615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:48.962898 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:48.962902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:48.962905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:48.962941 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:48.964493 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:48.965059 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:48.965063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:48.965065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:48.965146 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:48.965519 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:48.965524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:48.965526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:48.965738 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:48.965853 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:48.966378 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 06:55:48.966532 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:55:48.967653 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:48.953302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:44063)
0619 06:55:48.953927 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:48.953984 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:48.955430 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:48.956077 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:48.956086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:48.956494 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:48.957050 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:48.957063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.957958 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.957961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:48.959046 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:48.959056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:48.959671 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:48.959674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:48.959876 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:48.960763 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:48.960787 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:48.962703 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:48.964622 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:48.964629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:48.964646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:48.964947 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:48.965243 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:48.965247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:48.965249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:48.965420 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:48.965849 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:48.965856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:48.966136 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:55:48.966338 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 06:55:48.966691 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":146,"threads_started":1}
I20260619 06:55:48.969656 4475 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:48.970041 4475 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:55:48.970669 4475 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:48.972687 4284 ts_manager.cc:194] Registered new tserver with Master: 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:55:48.974223 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:44063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:55:48.989902 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:49.114456 4480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:49.114779 4480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:49.114859 4480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:49.119170 4480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:49.119335 4480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:49.119480 4480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:55:49.125176 4480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:49.126511 4480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:49.127869 4480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:49.139811 4488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:49.140156 4486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:49.139850 4485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:49.140283 4480 server_base.cc:1034] running on GCE node
I20260619 06:55:49.140887 4480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:49.141882 4480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:49.143116 4480 hybrid_clock.cc:648] HybridClock initialized: now 1781852149143062 us; error 72 us; skew 500 ppm
Jun 19 06:55:49 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:49.146464 4480 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:55:49.147727 4480 webserver.cc:460] Webserver started at http://127.22.113.2:33781/ using document root <none> and password file <none>
I20260619 06:55:49.148422 4480 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:49.148516 4480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:49.148799 4480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:49.151186 4480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "6ea9cc27f58aaf1662d28b55e0f5b8b4"
server_key_iv: "e20b6441605a71d5f77d23fff6f30646"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.151880 4480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "6ea9cc27f58aaf1662d28b55e0f5b8b4"
server_key_iv: "e20b6441605a71d5f77d23fff6f30646"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.157430 4480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:55:49.160423 4495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.161872 4480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:55:49.161975 4480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "6ea9cc27f58aaf1662d28b55e0f5b8b4"
server_key_iv: "e20b6441605a71d5f77d23fff6f30646"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.162073 4480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:49.185931 4480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:49.189739 4480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:49.189926 4480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:49.190549 4480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:49.191597 4480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:49.191648 4480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.191730 4480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:49.191764 4480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.203738 4480 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41127
I20260619 06:55:49.203810 4608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41127 every 8 connection(s)
I20260619 06:55:49.204833 4480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:55:49.208186 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4480
I20260619 06:55:49.208341 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:55:49.208703 22980 external_mini_cluster.cc:1428] Setting key 4483e60ddfa0853c48f8a17fcadf929e
Jun 19 06:55:49 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
I20260619 06:55:49.222767 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.207122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:38973 (local address 127.22.113.62:38627)
0619 06:55:49.207367 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.207373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.208571 (+ 1198us) server_negotiation.cc:407] Connection header received
0619 06:55:49.210462 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.210466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.210521 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.210601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:49.212545 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.213502 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.214735 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.215008 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.215099 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.218514 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:49.218520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:49.218525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:49.218565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:49.220398 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.221074 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.221081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.221085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.221179 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.221579 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.221582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.221583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.221794 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:49.221877 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.222511 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.222636 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:55:49.223940 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.207564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:38973)
0619 06:55:49.208275 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.208355 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.210207 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.210803 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.210812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.211416 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:49.212315 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.212341 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.213657 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.213660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.214606 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.214615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.215192 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.215199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.215465 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.216219 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:49.216246 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:49.218271 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:49.220576 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:49.220582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:49.220605 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:49.220920 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:49.221311 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:49.221314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:49.221316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:49.221466 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:49.221893 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:49.221899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:49.222210 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.222393 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.222718 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":141,"threads_started":1}
I20260619 06:55:49.225450 4609 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:49.225754 4609 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:49.226374 4609 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:49.227700 4284 ts_manager.cc:194] Registered new tserver with Master: 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127)
I20260619 06:55:49.228669 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:38973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:55:49.242774 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:49.361101 4614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:49.361390 4614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:49.361436 4614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:49.365492 4614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:49.365558 4614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:49.365619 4614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:55:49.371794 4614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:49.372925 4614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:49.374017 4614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:49.384774 4619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:49.384799 4622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:49.384770 4620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:49.385337 4614 server_base.cc:1034] running on GCE node
I20260619 06:55:49.385908 4614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:49.386847 4614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:49.388064 4614 hybrid_clock.cc:648] HybridClock initialized: now 1781852149388038 us; error 45 us; skew 500 ppm
Jun 19 06:55:49 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:49.391530 4614 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:55:49.392529 4614 webserver.cc:460] Webserver started at http://127.22.113.3:35849/ using document root <none> and password file <none>
I20260619 06:55:49.393131 4614 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:49.393193 4614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:49.393376 4614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:55:49.395637 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "015c641dcebb429ebe13198d7772e5a6"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "b5a8911bc95afd72ea3f5a355d85c437"
server_key_iv: "3d0b0ea01a5273942ab0f921ee5c31e9"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.396193 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "015c641dcebb429ebe13198d7772e5a6"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "b5a8911bc95afd72ea3f5a355d85c437"
server_key_iv: "3d0b0ea01a5273942ab0f921ee5c31e9"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.400429 4614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:55:49.403432 4629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.405011 4614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:49.405162 4614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "015c641dcebb429ebe13198d7772e5a6"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "b5a8911bc95afd72ea3f5a355d85c437"
server_key_iv: "3d0b0ea01a5273942ab0f921ee5c31e9"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.405295 4614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:49.418038 4614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:49.422775 4614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:49.423056 4614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:49.423890 4614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:49.424952 4614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:49.425011 4614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.425088 4614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:49.425133 4614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.436707 4614 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:33335
I20260619 06:55:49.436800 4742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:33335 every 8 connection(s)
I20260619 06:55:49.438004 4614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:55:49.440663 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4614
I20260619 06:55:49.440824 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:55:49.441190 22980 external_mini_cluster.cc:1428] Setting key 9f82bb31e370d758c015701f77afee1d
Jun 19 06:55:49 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:49.454250 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.440756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:39085 (local address 127.22.113.62:38627)
0619 06:55:49.440931 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.440935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.442495 (+ 1560us) server_negotiation.cc:407] Connection header received
0619 06:55:49.444325 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.444328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.444397 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.444474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:49.445727 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.446272 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.447482 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.447715 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.447826 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.450447 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:49.450450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:49.450452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:49.450475 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:49.451868 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.452567 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.452570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.452571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.452645 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.452991 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.452993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.452994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.453161 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:49.453242 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.454014 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.454124 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:55:49.455969 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.441627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:39085)
0619 06:55:49.442330 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.442377 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.444069 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.444668 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.444676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.445071 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:49.445577 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.445599 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.446399 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.446403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.447363 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.447372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.447869 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.447874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.448061 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.448717 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:49.448734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:49.450262 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:49.452027 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:49.452035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:49.452054 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:49.452458 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:49.452739 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:49.452742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:49.452743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:49.452900 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:49.453258 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:49.453264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:49.453756 (+ 492us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.453972 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.454467 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":266,"threads_started":1}
I20260619 06:55:49.457373 4743 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:49.457700 4743 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:49.458273 4743 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:49.459607 4284 ts_manager.cc:194] Registered new tserver with Master: 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
I20260619 06:55:49.460335 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:39085
I20260619 06:55:49.468518 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:55:49.478971 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.470712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.22.113.62:38627)
0619 06:55:49.470877 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.470884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.471066 (+ 182us) server_negotiation.cc:407] Connection header received
0619 06:55:49.471310 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.471313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.471358 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.471436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:49.472384 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.472922 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.473880 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.474110 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.474205 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.475974 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:49.475977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:49.475979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:49.476002 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:49.477365 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.477791 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.477793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.477794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.477865 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:49.478118 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:49.478120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:49.478121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:49.478346 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:49.478449 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.478694 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.478789 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:55:49.482024 4284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:55:49.485145 4284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:55:49.502147 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.22.113.1:34965)
0619 06:55:49.496676 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.496682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.496701 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:55:49.496781 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.496785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.496932 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.497063 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:49.497626 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.498863 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.500552 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.501272 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.501668 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.501828 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.501881 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.501977 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":141,"threads_started":1}
I20260619 06:55:49.502468 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.22.113.2:41127)
0619 06:55:49.496565 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.496576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.496595 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:55:49.496674 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.496680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.496893 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.497063 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:49.497636 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.498544 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.500380 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.501231 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.501366 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.501552 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.502152 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.502301 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":148,"threads_started":1}
I20260619 06:55:49.502261 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.0.0.1:46292)
0619 06:55:49.495942 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.496001 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.496141 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.497234 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.497238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.497257 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:49.497504 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.497511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.499006 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.499010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.500417 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.500427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.501419 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.501422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.501601 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.501724 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.501808 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.502035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":208,"thread_start_us":119,"threads_started":1}
I20260619 06:55:49.503077 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41127 (local address 127.0.0.1:42456)
0619 06:55:49.496199 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.496215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.496303 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.497091 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.497101 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.497135 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:49.497474 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.497487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.498700 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.498707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.500238 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.500249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.501392 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.501399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.501630 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.501695 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.501818 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.502002 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":153,"thread_start_us":113,"threads_started":1}
I20260619 06:55:49.503559 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.22.113.3:33335)
0619 06:55:49.496429 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.496444 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.496470 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:55:49.496717 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.496726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.497009 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.497209 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:49.497676 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.498863 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.500998 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.501963 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.502073 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.502211 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.503203 (+ 992us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.503388 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":210,"threads_started":1}
I20260619 06:55:49.503921 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.495056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33335 (local address 127.0.0.1:60478)
0619 06:55:49.496367 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.496385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.496492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.497329 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.497332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.497346 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:49.497521 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.497527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.499011 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.499014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.500840 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.500849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.502632 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.502636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.502758 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.502784 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.502856 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.503752 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"mutex_wait_us":906,"thread_start_us":149,"threads_started":1}
I20260619 06:55:49.506469 4409 tablet_service.cc:1467] Processing CreateTablet for tablet 925aabd50d074c279b2c276aa639aff5 (DEFAULT_TABLE table=test-table [id=b95b64652ceb44ff9456b8eef033a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:49.506711 4543 tablet_service.cc:1467] Processing CreateTablet for tablet 925aabd50d074c279b2c276aa639aff5 (DEFAULT_TABLE table=test-table [id=b95b64652ceb44ff9456b8eef033a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:49.506711 4677 tablet_service.cc:1467] Processing CreateTablet for tablet 925aabd50d074c279b2c276aa639aff5 (DEFAULT_TABLE table=test-table [id=b95b64652ceb44ff9456b8eef033a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:49.508024 4409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925aabd50d074c279b2c276aa639aff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:49.508028 4677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925aabd50d074c279b2c276aa639aff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:49.508026 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925aabd50d074c279b2c276aa639aff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:49.515867 4761 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Bootstrap starting.
I20260619 06:55:49.516422 4760 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Bootstrap starting.
I20260619 06:55:49.517282 4762 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap starting.
I20260619 06:55:49.520161 4762 tablet_bootstrap.cc:654] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:49.520702 4761 tablet_bootstrap.cc:654] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:49.521071 4760 tablet_bootstrap.cc:654] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:49.521332 4762 log.cc:824] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:49.521539 4761 log.cc:824] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:49.522521 4760 log.cc:824] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:49.524034 4761 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: No bootstrap required, opened a new log
I20260619 06:55:49.524041 4762 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: No bootstrap required, opened a new log
I20260619 06:55:49.524281 4761 ts_tablet_manager.cc:1397] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:55:49.524279 4762 ts_tablet_manager.cc:1397] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:55:49.524832 4760 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: No bootstrap required, opened a new log
I20260619 06:55:49.525058 4760 ts_tablet_manager.cc:1397] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:55:49.528133 4760 raft_consensus.cc:348] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.528393 4760 raft_consensus.cc:374] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:49.528446 4760 raft_consensus.cc:729] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015c641dcebb429ebe13198d7772e5a6, State: Initialized, Role: FOLLOWER
I20260619 06:55:49.528898 4760 consensus_queue.cc:260] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.529161 4761 raft_consensus.cc:348] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.529472 4761 raft_consensus.cc:374] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:49.529542 4761 raft_consensus.cc:729] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9f39ce2d9c4c4e9dd0c1551376b741, State: Initialized, Role: FOLLOWER
I20260619 06:55:49.529644 4743 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
I20260619 06:55:49.529522 4762 raft_consensus.cc:348] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.529760 4762 raft_consensus.cc:374] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:49.529814 4762 raft_consensus.cc:729] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e3e6391d74f73a9eabeaddc832bb7, State: Initialized, Role: FOLLOWER
I20260619 06:55:49.529958 4760 ts_tablet_manager.cc:1428] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:55:49.530112 4761 consensus_queue.cc:260] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.530229 4762 consensus_queue.cc:260] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.530920 4609 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
I20260619 06:55:49.531229 4761 ts_tablet_manager.cc:1428] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:55:49.531785 4475 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
I20260619 06:55:49.532881 4762 ts_tablet_manager.cc:1428] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:55:49.593886 4767 raft_consensus.cc:482] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:49.594182 4767 raft_consensus.cc:504] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.595414 4767 leader_election.cc:288] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335), 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
I20260619 06:55:49.599978 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.595828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:56975 (local address 127.22.113.3:33335)
0619 06:55:49.595982 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.595987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.596006 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:55:49.596175 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.596183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.596243 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.596352 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:49.596846 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.597476 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.598675 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.599203 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.599416 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.599461 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.599748 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.599829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:55:49.599980 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.595740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33335 (local address 127.22.113.2:56975)
0619 06:55:49.595891 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.595909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.596014 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.596403 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.596406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.596425 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:49.596651 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.596692 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.597589 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.597593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.598539 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.598549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.599427 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.599431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.599556 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.599588 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.599728 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.599821 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"spinlock_wait_cycles":9984}
I20260619 06:55:49.600499 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.595740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.22.113.2:38119)
0619 06:55:49.596366 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.596384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:49.596535 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:49.596847 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:49.596850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:49.596867 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:49.597032 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.597039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.597906 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.597908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.598890 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:49.598900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.599785 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.599789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:49.599930 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.599957 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:55:49.600248 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 06:55:49.600335 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":154,"spinlock_wait_cycles":3200,"thread_start_us":186,"threads_started":1}
I20260619 06:55:49.600605 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.595840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:38119 (local address 127.22.113.1:34965)
0619 06:55:49.595982 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.595987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.596508 (+ 521us) server_negotiation.cc:407] Connection header received
0619 06:55:49.596655 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.596659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.596728 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.596818 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:49.597154 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.597777 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.599024 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.599645 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.599767 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.599820 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.600313 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.600465 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:55:49.600914 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925aabd50d074c279b2c276aa639aff5" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015c641dcebb429ebe13198d7772e5a6" is_pre_election: true
I20260619 06:55:49.601327 4697 raft_consensus.cc:2454] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 0.
I20260619 06:55:49.601349 4429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925aabd50d074c279b2c276aa639aff5" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e3e6391d74f73a9eabeaddc832bb7" is_pre_election: true
I20260619 06:55:49.601691 4429 raft_consensus.cc:2454] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 0.
I20260619 06:55:49.601926 4499 leader_election.cc:302] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741; no voters:
I20260619 06:55:49.602252 4767 raft_consensus.cc:2781] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:55:49.602340 4767 raft_consensus.cc:482] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:55:49.602393 4767 raft_consensus.cc:3037] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:49.603499 4767 raft_consensus.cc:504] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.603859 4767 leader_election.cc:288] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 election: Requested vote from peers 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335), 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
I20260619 06:55:49.604312 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925aabd50d074c279b2c276aa639aff5" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015c641dcebb429ebe13198d7772e5a6"
I20260619 06:55:49.604351 4429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925aabd50d074c279b2c276aa639aff5" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e3e6391d74f73a9eabeaddc832bb7"
I20260619 06:55:49.604485 4429 raft_consensus.cc:3037] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:49.604480 4697 raft_consensus.cc:3037] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:49.605657 4429 raft_consensus.cc:2454] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 1.
I20260619 06:55:49.605898 4697 raft_consensus.cc:2454] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 1.
I20260619 06:55:49.605970 4498 leader_election.cc:302] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7, 3d9f39ce2d9c4c4e9dd0c1551376b741; no voters:
I20260619 06:55:49.606168 4767 raft_consensus.cc:2781] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:49.606464 4767 raft_consensus.cc:686] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 LEADER]: Becoming Leader. State: Replica: 3d9f39ce2d9c4c4e9dd0c1551376b741, State: Running, Role: LEADER
I20260619 06:55:49.606807 4767 consensus_queue.cc:237] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:49.610431 4282 catalog_manager.cc:5495] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } health_report { overall_health: HEALTHY } } }
I20260619 06:55:49.627816 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.624197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.22.113.2:41127)
0619 06:55:49.624376 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:49.624389 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:49.624409 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:55:49.624485 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:49.624487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:49.624545 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:49.624666 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:55:49.624944 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.625458 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.626283 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:49.626502 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:49.626609 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:49.626916 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:55:49.627433 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:55:49.627565 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:49.627606 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 06:55:49.627661 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:55:49.635139 4429 raft_consensus.cc:1261] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Refusing update from remote peer 3d9f39ce2d9c4c4e9dd0c1551376b741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:49.635411 4697 raft_consensus.cc:1261] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Refusing update from remote peer 3d9f39ce2d9c4c4e9dd0c1551376b741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:49.636049 4770 consensus_queue.cc:1035] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:49.636250 4767 consensus_queue.cc:1035] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:49.645449 4775 mvcc.cc:205] Tried to move back new op lower bound from 7298466404896968704 to 7298466404792463360. Current Snapshot: MvccSnapshot[applied={T|T < 7298466404896968704}]
I20260619 06:55:49.647264 4774 mvcc.cc:205] Tried to move back new op lower bound from 7298466404896968704 to 7298466404792463360. Current Snapshot: MvccSnapshot[applied={T|T < 7298466404896968704}]
I20260619 06:55:49.656401 4283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:49.656618 4283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:49.659874 4283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Sending DeleteTablet for 3 replicas of tablet 925aabd50d074c279b2c276aa639aff5
I20260619 06:55:49.660986 4409 tablet_service.cc:1514] Processing DeleteTablet for tablet 925aabd50d074c279b2c276aa639aff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:46292
I20260619 06:55:49.660979 4677 tablet_service.cc:1514] Processing DeleteTablet for tablet 925aabd50d074c279b2c276aa639aff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60478
I20260619 06:55:49.661067 4543 tablet_service.cc:1514] Processing DeleteTablet for tablet 925aabd50d074c279b2c276aa639aff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260619 06:55:49.661315 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4346
I20260619 06:55:49.662168 4784 tablet_replica.cc:330] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: stopping tablet replica
I20260619 06:55:49.662622 4784 raft_consensus.cc:2229] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:55:49.663007 4783 tablet_replica.cc:330] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: stopping tablet replica
I20260619 06:55:49.663053 4784 raft_consensus.cc:2258] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:49.663331 4783 raft_consensus.cc:2229] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:55:49.663793 4783 raft_consensus.cc:2258] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:49.665251 4784 ts_tablet_manager.cc:1905] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:49.665251 4783 ts_tablet_manager.cc:1905] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:49.668571 4783 ts_tablet_manager.cc:1918] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:49.668571 4784 ts_tablet_manager.cc:1918] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:49.668694 4783 log.cc:1196] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/925aabd50d074c279b2c276aa639aff5
I20260619 06:55:49.668840 4784 log.cc:1196] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/925aabd50d074c279b2c276aa639aff5
I20260619 06:55:49.669066 4783 ts_tablet_manager.cc:1939] T 925aabd50d074c279b2c276aa639aff5 P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting consensus metadata
I20260619 06:55:49.669338 4784 ts_tablet_manager.cc:1939] T 925aabd50d074c279b2c276aa639aff5 P 015c641dcebb429ebe13198d7772e5a6: Deleting consensus metadata
W20260619 06:55:49.669469 4270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:34965 (error 108)
I20260619 06:55:49.670382 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4480
I20260619 06:55:49.672184 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.671755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.0.0.1:46304)
0619 06:55:49.671936 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.672055 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:55:49.672453 4271 catalog_manager.cc:4841] TS 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): tablet 925aabd50d074c279b2c276aa639aff5 (table test-table [id=b95b64652ceb44ff9456b8eef033a918]) successfully deleted
W20260619 06:55:49.672550 4270 catalog_manager.cc:4584] TS 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 925aabd50d074c279b2c276aa639aff5: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34965: connect: Connection refused (error 111)
I20260619 06:55:49.677448 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4614
I20260619 06:55:49.678654 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:49.677680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41127 (local address 127.0.0.1:42472)
0619 06:55:49.678422 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:49.678497 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635}
I20260619 06:55:49.683996 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4252
I20260619 06:55:49.691655 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--webserver_interface=127.22.113.62
--webserver_port=35207
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:49.820547 4786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:49.820863 4786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:49.820909 4786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:49.825210 4786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:49.825284 4786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:49.825305 4786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:49.825325 4786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:49.825341 4786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:49.831121 4786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=35207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:49.832513 4786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:49.833783 4786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:49.849571 4795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:49.849632 4792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:49.849838 4786 server_base.cc:1034] running on GCE node
W20260619 06:55:49.849617 4793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:49.850682 4786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:49.852165 4786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:49.853330 4786 hybrid_clock.cc:648] HybridClock initialized: now 1781852149853308 us; error 41 us; skew 500 ppm
Jun 19 06:55:49 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:49.856725 4786 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:49.857831 4786 webserver.cc:460] Webserver started at http://127.22.113.62:35207/ using document root <none> and password file <none>
I20260619 06:55:49.858566 4786 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:49.858686 4786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:49.863430 4786 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:49.866024 4802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:49.867643 4786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:55:49.867800 4786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "094abe28bdc24678b8c74b074a1b01cd"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "b37773f704ae5b854858288d5a7bb0c2"
server_key_iv: "3cb1597b74715e95646e1d24b45e07b8"
server_key_version: "encryptionkey@0"
I20260619 06:55:49.868516 4786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:49.893311 4786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:49.897423 4786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:49.897679 4786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:49.906781 4786 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:38627
I20260619 06:55:49.907133 4854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:38627 every 8 connection(s)
I20260619 06:55:49.908200 4786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:49.909920 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4786
I20260619 06:55:49.910499 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:34965
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=32857
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:49.914038 4855 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:55:49.915961 4855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap starting.
I20260619 06:55:49.933384 4855 log.cc:824] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:49.942337 4855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:55:49.942778 4855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap complete.
I20260619 06:55:49.947233 4855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:49.947758 4855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Initialized, Role: FOLLOWER
I20260619 06:55:49.948340 4855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:49.948465 4855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:49.948565 4855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:49.948700 4855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:55:49.950831 4855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:49.951207 4855 leader_election.cc:302] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094abe28bdc24678b8c74b074a1b01cd; no voters:
I20260619 06:55:49.951647 4855 leader_election.cc:288] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:55:49.951736 4859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:55:49.951946 4859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 LEADER]: Becoming Leader. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Running, Role: LEADER
I20260619 06:55:49.952268 4859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:49.953023 4855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:49.955651 4860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:49.955780 4860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:49.956050 4861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094abe28bdc24678b8c74b074a1b01cd. Latest consensus state: current_term: 2 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:49.956139 4861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:49.958540 4874 catalog_manager.cc:1259] Loaded cluster ID: 96ba8d67202e48f0a4664b996a6e1894
I20260619 06:55:49.958621 4874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: loading cluster ID for follower catalog manager: success
W20260619 06:55:49.959571 4874 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260619 06:55:49.959648 4874 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260619 06:55:49.959728 4875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:49.961755 4875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:49.962198 4875 catalog_manager.cc:1259] Loaded cluster ID: 96ba8d67202e48f0a4664b996a6e1894
I20260619 06:55:49.962268 4875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:55:49.963100 4875 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 06:55:49.966606 4875 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 06:55:49.971961 4875 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:55:49.972810 4875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Loaded TSK: 0
I20260619 06:55:49.974010 4875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:55:50.065764 4857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:50.066085 4857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:50.066166 4857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:50.070550 4857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:50.070636 4857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:50.070724 4857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:55:50.076495 4857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:34965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=32857
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:50.077665 4857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:50.078851 4857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:50.088743 4883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.088743 4885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.088802 4882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:50.090322 4857 server_base.cc:1034] running on GCE node
I20260619 06:55:50.090765 4857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:50.091550 4857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:50.092749 4857 hybrid_clock.cc:648] HybridClock initialized: now 1781852150092714 us; error 58 us; skew 500 ppm
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:50.095744 4857 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:55:50.096729 4857 webserver.cc:460] Webserver started at http://127.22.113.1:32857/ using document root <none> and password file <none>
I20260619 06:55:50.097345 4857 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:50.097422 4857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:50.101831 4857 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:55:50.104408 4892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.106030 4857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:50.106227 4857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "229e3e6391d74f73a9eabeaddc832bb7"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "765762330a28a59aff1abf1854799a46"
server_key_iv: "779d041c57b0ea40070a552297bc1adb"
server_key_version: "encryptionkey@0"
I20260619 06:55:50.106799 4857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:50.124541 4857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:50.128197 4857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:50.128422 4857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:50.129078 4857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:50.130255 4899 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:55:50.136883 4857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:55:50.137005 4857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260619 06:55:50.137105 4857 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:55:50.139075 4857 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:55:50.139150 4857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 06:55:50.139660 4899 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap starting.
I20260619 06:55:50.150708 4857 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:34965
I20260619 06:55:50.150956 5006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:34965 every 8 connection(s)
I20260619 06:55:50.151929 4857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:55:50.160365 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 4857
I20260619 06:55:50.160869 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:41127
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=33781
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:50.164238 4899 log.cc:824] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:50.174307 4899 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:55:50.174805 4899 tablet_bootstrap.cc:492] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap complete.
I20260619 06:55:50.175770 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.154251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:52147 (local address 127.22.113.62:38627)
0619 06:55:50.154798 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.154825 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.155402 (+ 577us) server_negotiation.cc:407] Connection header received
0619 06:55:50.157263 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.157281 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.157700 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.158106 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:50.160125 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.162004 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.163980 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.164605 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.164800 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.168456 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:50.168466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:50.168495 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:50.168544 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:50.171286 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.172025 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.172033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.172041 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.172229 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.172774 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.172780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.172784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.173379 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:50.173616 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.174382 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.174760 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":169,"threads_started":1}
I20260619 06:55:50.176194 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.154381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:52147)
0619 06:55:50.155126 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.155204 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.156709 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.158600 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.158610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.159156 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:50.159878 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.159922 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.162235 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.162240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.163772 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.163790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.164785 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.164790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.165037 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.165854 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:50.165877 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:50.168143 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:50.171432 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.171439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.171452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.171853 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.172369 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.172373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.172375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.172581 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.173543 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:50.173549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:50.173893 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.174160 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.174691 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":168,"threads_started":1}
I20260619 06:55:50.176367 4899 ts_tablet_manager.cc:1397] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.008s
I20260619 06:55:50.179184 5007 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:50.179522 5007 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:50.179420 4899 raft_consensus.cc:348] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:50.179843 4899 raft_consensus.cc:729] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e3e6391d74f73a9eabeaddc832bb7, State: Initialized, Role: FOLLOWER
I20260619 06:55:50.180140 5007 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:50.180320 4899 consensus_queue.cc:260] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } } peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } }
I20260619 06:55:50.181363 4899 ts_tablet_manager.cc:1428] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:55:50.183390 4819 ts_manager.cc:194] Registered new tserver with Master: 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
I20260619 06:55:50.187587 4819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:52147
I20260619 06:55:50.190686 5007 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852149, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:55:50.198334 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.186712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.22.113.1:34965)
0619 06:55:50.187687 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.187700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.187730 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:55:50.187815 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.187821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.188037 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.188201 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:50.189021 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.189807 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.190722 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.191314 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.191435 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.194374 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:50.194384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:50.194407 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:50.194432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:50.196233 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.196750 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.196756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.196760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.196884 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.197183 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.197188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.197190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.197459 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:50.197610 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.197969 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.198165 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":141,"threads_started":1}
I20260619 06:55:50.198334 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.186650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.0.0.1:46306)
0619 06:55:50.187117 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.187175 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.187346 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.188233 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.188239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.188734 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:50.188881 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.188888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.189931 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.189937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.190587 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.190596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.191443 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.191447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.191568 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.192260 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:50.192273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:50.194214 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:50.196373 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.196382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.196388 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.196628 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.196976 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.196979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.196981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.197079 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.197578 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:50.197585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:50.197724 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.197873 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.198113 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260619 06:55:50.199973 4941 tablet_service.cc:1514] Processing DeleteTablet for tablet 925aabd50d074c279b2c276aa639aff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:49 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:46306
I20260619 06:55:50.201330 5016 tablet_replica.cc:330] stopping tablet replica
I20260619 06:55:50.201740 5016 raft_consensus.cc:2229] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:55:50.202049 5016 raft_consensus.cc:2258] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:50.203286 5016 ts_tablet_manager.cc:1905] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:50.207489 5016 ts_tablet_manager.cc:1918] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:50.207604 5016 log.cc:1196] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/925aabd50d074c279b2c276aa639aff5
I20260619 06:55:50.207897 5016 ts_tablet_manager.cc:1939] T 925aabd50d074c279b2c276aa639aff5 P 229e3e6391d74f73a9eabeaddc832bb7: Deleting consensus metadata
I20260619 06:55:50.209969 4805 catalog_manager.cc:4841] TS 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965): tablet 925aabd50d074c279b2c276aa639aff5 (table test-table [id=b95b64652ceb44ff9456b8eef033a918]) successfully deleted
W20260619 06:55:50.290967 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:50.291297 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:50.291357 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:50.295395 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:50.295454 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:50.295512 5011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:55:50.300690 5011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:41127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=33781
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:50.301723 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:50.302695 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:50.313908 5021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.313977 5022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.313908 5024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:50.314257 5011 server_base.cc:1034] running on GCE node
I20260619 06:55:50.314774 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:50.315541 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:50.316712 5011 hybrid_clock.cc:648] HybridClock initialized: now 1781852150316681 us; error 56 us; skew 500 ppm
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:50.319716 5011 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:55:50.320642 5011 webserver.cc:460] Webserver started at http://127.22.113.2:33781/ using document root <none> and password file <none>
I20260619 06:55:50.321169 5011 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:50.321244 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:50.325300 5011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:55:50.327562 5031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.328960 5011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:55:50.329100 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "6ea9cc27f58aaf1662d28b55e0f5b8b4"
server_key_iv: "e20b6441605a71d5f77d23fff6f30646"
server_key_version: "encryptionkey@0"
I20260619 06:55:50.329668 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:50.342036 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:50.345103 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:50.345261 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:50.345803 5011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:50.346673 5011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:50.346724 5011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.346800 5011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:50.346841 5011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.358008 5011 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41127
I20260619 06:55:50.358028 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41127 every 8 connection(s)
I20260619 06:55:50.359102 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:55:50.359925 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5011
I20260619 06:55:50.360489 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:33335
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=35849
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:50.377496 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.361655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:44657 (local address 127.22.113.62:38627)
0619 06:55:50.361884 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.361892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.362705 (+ 813us) server_negotiation.cc:407] Connection header received
0619 06:55:50.364562 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.364567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.364632 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.364743 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:50.366417 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.367018 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.368994 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.369263 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.369439 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.372968 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:50.372972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:50.372976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:50.373073 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:50.374862 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.375597 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.375601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.375604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.375691 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.376086 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.376089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.376091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.376393 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:50.376509 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.377136 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.377292 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:55:50.378568 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.362038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:44657)
0619 06:55:50.362535 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.362598 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.364284 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.364964 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.364975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.365489 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:50.366222 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.366246 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.367171 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.367176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.368847 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.368863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.369501 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.369507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.369759 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.370515 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:50.370543 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:50.372713 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:50.375009 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.375017 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.375039 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.375463 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.375794 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.375800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.375802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.375968 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.376517 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:50.376527 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:50.376849 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.377073 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.377489 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260619 06:55:50.380124 5145 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:50.380453 5145 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:50.381093 5145 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:50.382319 4819 ts_manager.cc:194] Registered new tserver with Master: 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127)
I20260619 06:55:50.382992 4819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:44657
W20260619 06:55:50.477365 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:50.477648 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:50.477692 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:50.481793 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:50.481865 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:50.481956 5147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:55:50.487532 5147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:33335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=35849
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:50.488616 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:50.489620 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:50.500077 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:50.500213 5147 server_base.cc:1034] running on GCE node
W20260619 06:55:50.500061 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.500061 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:50.500946 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:50.501718 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:50.502897 5147 hybrid_clock.cc:648] HybridClock initialized: now 1781852150502852 us; error 56 us; skew 500 ppm
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:50.506166 5147 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:55:50.507162 5147 webserver.cc:460] Webserver started at http://127.22.113.3:35849/ using document root <none> and password file <none>
I20260619 06:55:50.507849 5147 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:50.507931 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:50.512878 5147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:55:50.515543 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.517076 5147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:50.517253 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "015c641dcebb429ebe13198d7772e5a6"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "b5a8911bc95afd72ea3f5a355d85c437"
server_key_iv: "3d0b0ea01a5273942ab0f921ee5c31e9"
server_key_version: "encryptionkey@0"
I20260619 06:55:50.517897 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:50.546324 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:50.550328 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:50.550561 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:50.551311 5147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:50.552281 5147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:50.552361 5147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.552452 5147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:50.552506 5147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.563887 5147 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:33335
I20260619 06:55:50.563920 5276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:33335 every 8 connection(s)
I20260619 06:55:50.565038 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:55:50.569579 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5147
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:55:50.580518 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.570841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.22.113.62:38627)
0619 06:55:50.571130 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.571135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.571151 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:55:50.571194 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.571199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.571271 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.571376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:50.572249 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.573013 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.574168 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.574542 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.574657 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.576291 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:50.576297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:50.576304 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:50.576331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:50.578281 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.578805 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.578810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.578813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.578935 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.579319 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.579322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.579324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.579632 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:50.579823 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.580087 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.580248 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260619 06:55:50.582916 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.567234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:56735 (local address 127.22.113.62:38627)
0619 06:55:50.567481 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.567494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.568581 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 06:55:50.570560 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.570564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.570637 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.570742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:50.572506 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.573222 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.574898 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.575175 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.575332 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.578435 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:50.578439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:50.578442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:50.578477 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:50.580436 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.581046 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.581051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.581053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.581140 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:50.581585 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:50.581588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:50.581590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:50.581820 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:50.581922 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.582673 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.582789 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:55:50.583787 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.567778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:56735)
0619 06:55:50.568352 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.568407 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.570302 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.571011 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.571023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.571639 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:50.572268 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.572292 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.573400 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.573405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.574736 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.574748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.575391 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.575397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.575601 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.576187 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:50.576236 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:50.578171 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:50.580546 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.580553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.580569 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.580917 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.581261 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:50.581264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:50.581266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:50.581472 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:50.581936 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:50.581942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:50.582204 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.582417 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.582759 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":133,"threads_started":1}
I20260619 06:55:50.585038 5277 heartbeater.cc:347] Connected to a master server at 127.22.113.62:38627
I20260619 06:55:50.585353 5277 heartbeater.cc:464] Registering TS with master...
I20260619 06:55:50.585956 5277 heartbeater.cc:511] Master 127.22.113.62:38627 requested a full tablet report, sending...
I20260619 06:55:50.587077 4819 ts_manager.cc:194] Registered new tserver with Master: 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
I20260619 06:55:50.587695 4819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:56735
I20260619 06:55:50.594487 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:55:50.599213 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.595176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.22.113.62:38627)
0619 06:55:50.595364 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.595367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.595682 (+ 315us) server_negotiation.cc:407] Connection header received
0619 06:55:50.595944 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.595947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.596052 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.596155 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:55:50.596410 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.596932 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.597836 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.598083 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.598177 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.598474 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:55:50.598866 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:55:50.598990 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.599059 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.599105 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 06:55:50.599932 4819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:55:50.603307 4819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:55:50.611907 4941 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae12c1847b440aa59a2edcd930222c (DEFAULT_TABLE table=test-table [id=e43947cd5eb249499d76be899ca2aef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:50.612738 4941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae12c1847b440aa59a2edcd930222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:50.616325 4899 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap starting.
I20260619 06:55:50.617022 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.611064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33335 (local address 127.0.0.1:60484)
0619 06:55:50.611449 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.611466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.611558 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.612381 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.612385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.612408 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:50.612547 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.612554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.614301 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.614304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.615414 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.615428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.616550 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.616554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.616646 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.616701 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.616758 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.616838 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":110,"thread_start_us":196,"threads_started":1}
I20260619 06:55:50.617396 4899 tablet_bootstrap.cc:654] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:50.617449 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.610718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41127 (local address 127.0.0.1:42484)
0619 06:55:50.610897 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.610914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.610993 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.612232 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.612235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.612252 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:50.612378 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.612384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.614250 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.614253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.615619 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.615631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.616948 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.616951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.617054 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.617085 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.617167 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.617270 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1}
I20260619 06:55:50.617584 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.611577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.22.113.3:33335)
0619 06:55:50.611980 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.611991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.612010 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:55:50.612081 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.612086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.612243 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.612407 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:50.612683 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.614105 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.615558 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.616404 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.617172 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.617281 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.617325 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.617404 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":166,"threads_started":1}
I20260619 06:55:50.617975 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.610939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.22.113.2:41127)
0619 06:55:50.611661 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.611675 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.611698 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:55:50.611810 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.611815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.612058 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.612515 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:50.612682 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.614105 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.615769 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.616822 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.616928 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.617072 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.617783 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.617859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":150,"threads_started":1}
I20260619 06:55:50.620224 4899 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: No bootstrap required, opened a new log
I20260619 06:55:50.620353 4899 ts_tablet_manager.cc:1397] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:55:50.620908 4899 raft_consensus.cc:348] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.621083 4899 raft_consensus.cc:374] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:50.621124 4899 raft_consensus.cc:729] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e3e6391d74f73a9eabeaddc832bb7, State: Initialized, Role: FOLLOWER
I20260619 06:55:50.621272 4899 consensus_queue.cc:260] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.621562 4899 ts_tablet_manager.cc:1428] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:55:50.621199 5211 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae12c1847b440aa59a2edcd930222c (DEFAULT_TABLE table=test-table [id=e43947cd5eb249499d76be899ca2aef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:50.621399 5079 tablet_service.cc:1467] Processing CreateTablet for tablet 6eae12c1847b440aa59a2edcd930222c (DEFAULT_TABLE table=test-table [id=e43947cd5eb249499d76be899ca2aef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:55:50.622431 5211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae12c1847b440aa59a2edcd930222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:50.622773 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eae12c1847b440aa59a2edcd930222c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:55:50.630030 5290 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Bootstrap starting.
I20260619 06:55:50.631312 5291 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Bootstrap starting.
I20260619 06:55:50.632589 5290 tablet_bootstrap.cc:654] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:50.633822 5291 tablet_bootstrap.cc:654] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Neither blocks nor log segments found. Creating new log.
I20260619 06:55:50.634047 5290 log.cc:824] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:50.634613 5291 log.cc:824] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:50.636447 5290 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: No bootstrap required, opened a new log
I20260619 06:55:50.636689 5290 ts_tablet_manager.cc:1397] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:55:50.636873 5291 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: No bootstrap required, opened a new log
I20260619 06:55:50.637106 5291 ts_tablet_manager.cc:1397] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:55:50.640148 5291 raft_consensus.cc:348] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.640187 5290 raft_consensus.cc:348] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.640414 5291 raft_consensus.cc:374] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:50.640451 5290 raft_consensus.cc:374] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:55:50.640498 5290 raft_consensus.cc:729] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015c641dcebb429ebe13198d7772e5a6, State: Initialized, Role: FOLLOWER
I20260619 06:55:50.640508 5291 raft_consensus.cc:729] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9f39ce2d9c4c4e9dd0c1551376b741, State: Initialized, Role: FOLLOWER
I20260619 06:55:50.640993 5291 consensus_queue.cc:260] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.641031 5290 consensus_queue.cc:260] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.641690 5145 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
I20260619 06:55:50.641742 5277 heartbeater.cc:503] Master 127.22.113.62:38627 was elected leader, sending a full tablet report...
I20260619 06:55:50.642061 5291 ts_tablet_manager.cc:1428] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:55:50.642140 5290 ts_tablet_manager.cc:1428] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 06:55:50.653353 5008 tablet.cc:2369] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:55:50.714612 5294 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:50.714824 5294 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.716055 5294 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335), 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
I20260619 06:55:50.720047 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.716254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33335 (local address 127.22.113.2:56457)
0619 06:55:50.716417 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.716434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.716533 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.716957 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.716961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.716985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:50.717127 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.717133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.717898 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.717904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.718763 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.718772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.719543 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.719547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.719675 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.719695 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.719796 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.719887 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:55:50.720047 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.716460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:56457 (local address 127.22.113.3:33335)
0619 06:55:50.716640 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.716645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.716666 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:55:50.716745 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.716748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.716809 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.716928 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:50.717239 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.717785 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.718901 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.719422 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.719551 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.719616 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.719814 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.719893 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:55:50.720587 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.716455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.22.113.2:39515)
0619 06:55:50.716917 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.716934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:50.717077 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:50.717376 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:50.717379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:50.717395 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:55:50.717544 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.717550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.718372 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.718377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.719353 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:50.719363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.720084 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.720088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:50.720197 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.720222 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:55:50.720333 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:55:50.720413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":193,"threads_started":1}
I20260619 06:55:50.720876 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.716560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:39515 (local address 127.22.113.1:34965)
0619 06:55:50.716916 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.716921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.717026 (+ 105us) server_negotiation.cc:407] Connection header received
0619 06:55:50.717186 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.717189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.717252 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.717367 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:55:50.717653 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.718256 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.719471 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.719936 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.720084 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.720245 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.720584 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.720679 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":154,"threads_started":1}
I20260619 06:55:50.720996 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae12c1847b440aa59a2edcd930222c" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015c641dcebb429ebe13198d7772e5a6" is_pre_election: true
I20260619 06:55:50.721400 5231 raft_consensus.cc:2454] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 0.
I20260619 06:55:50.721396 4961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae12c1847b440aa59a2edcd930222c" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e3e6391d74f73a9eabeaddc832bb7" is_pre_election: true
I20260619 06:55:50.721741 4961 raft_consensus.cc:2454] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 0.
I20260619 06:55:50.722141 5035 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741; no voters:
I20260619 06:55:50.722532 5294 raft_consensus.cc:2781] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:55:50.722651 5294 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:55:50.722712 5294 raft_consensus.cc:3037] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:50.723990 5294 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.724411 5294 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 election: Requested vote from peers 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335), 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965)
I20260619 06:55:50.724786 4961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae12c1847b440aa59a2edcd930222c" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e3e6391d74f73a9eabeaddc832bb7"
I20260619 06:55:50.724925 4961 raft_consensus.cc:3037] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:50.725039 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eae12c1847b440aa59a2edcd930222c" candidate_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015c641dcebb429ebe13198d7772e5a6"
I20260619 06:55:50.725205 5231 raft_consensus.cc:3037] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:55:50.726071 4961 raft_consensus.cc:2454] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 1.
I20260619 06:55:50.726361 5034 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7, 3d9f39ce2d9c4c4e9dd0c1551376b741; no voters:
I20260619 06:55:50.726569 5294 raft_consensus.cc:2781] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:55:50.726649 5231 raft_consensus.cc:2454] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9f39ce2d9c4c4e9dd0c1551376b741 in term 1.
I20260619 06:55:50.726854 5294 raft_consensus.cc:686] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 LEADER]: Becoming Leader. State: Replica: 3d9f39ce2d9c4c4e9dd0c1551376b741, State: Running, Role: LEADER
I20260619 06:55:50.727289 5294 consensus_queue.cc:237] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:50.730793 4815 catalog_manager.cc:5495] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } health_report { overall_health: UNKNOWN } } }
I20260619 06:55:50.745306 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.741628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.22.113.2:41127)
0619 06:55:50.741803 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:50.741809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:50.741843 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:55:50.741925 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:50.741928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:50.741994 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:50.742102 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:55:50.742358 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.742904 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.743835 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:50.744036 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:50.744146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:50.744475 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:55:50.744899 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:55:50.745036 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:50.745094 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:55:50.745151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:55:50.752554 4961 raft_consensus.cc:1261] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Refusing update from remote peer 3d9f39ce2d9c4c4e9dd0c1551376b741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:50.752581 5231 raft_consensus.cc:1261] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Refusing update from remote peer 3d9f39ce2d9c4c4e9dd0c1551376b741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:55:50.753413 5298 consensus_queue.cc:1035] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:50.753731 5294 consensus_queue.cc:1035] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:55:50.760994 5301 mvcc.cc:205] Tried to move back new op lower bound from 7298466409473347584 to 7298466409381658624. Current Snapshot: MvccSnapshot[applied={T|T < 7298466409473347584}]
I20260619 06:55:50.771190 4819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:50.771447 4819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:55:50.775617 4819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Sending DeleteTablet for 3 replicas of tablet 6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:50.776449 5079 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae12c1847b440aa59a2edcd930222c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:50 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:42484
I20260619 06:55:50.776539 5211 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eae12c1847b440aa59a2edcd930222c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:55:50 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60484
I20260619 06:55:50.776826 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4857
I20260619 06:55:50.777849 5310 tablet_replica.cc:330] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: stopping tablet replica
I20260619 06:55:50.778265 5310 raft_consensus.cc:2229] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:55:50.778409 5311 tablet_replica.cc:330] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: stopping tablet replica
I20260619 06:55:50.778841 5311 raft_consensus.cc:2229] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:55:50.778867 5310 raft_consensus.cc:2258] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:50.779217 5311 raft_consensus.cc:2258] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:55:50.781044 5310 ts_tablet_manager.cc:1905] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:50.781136 5311 ts_tablet_manager.cc:1905] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:55:50.784385 5310 ts_tablet_manager.cc:1918] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:50.784587 5310 log.cc:1196] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:50.784936 5311 ts_tablet_manager.cc:1918] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:55:50.785068 5311 log.cc:1196] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:50.785182 5310 ts_tablet_manager.cc:1939] T 6eae12c1847b440aa59a2edcd930222c P 3d9f39ce2d9c4c4e9dd0c1551376b741: Deleting consensus metadata
I20260619 06:55:50.785488 5311 ts_tablet_manager.cc:1939] T 6eae12c1847b440aa59a2edcd930222c P 015c641dcebb429ebe13198d7772e5a6: Deleting consensus metadata
W20260619 06:55:50.785513 4805 connection.cc:535] client connection to 127.22.113.1:34965 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 06:55:50.785657 4805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:55:50.786734 4804 catalog_manager.cc:4841] TS 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): tablet 6eae12c1847b440aa59a2edcd930222c (table test-table [id=e43947cd5eb249499d76be899ca2aef2]) successfully deleted
I20260619 06:55:50.786852 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5011
I20260619 06:55:50.788000 4806 catalog_manager.cc:4841] TS 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): tablet 6eae12c1847b440aa59a2edcd930222c (table test-table [id=e43947cd5eb249499d76be899ca2aef2]) successfully deleted
I20260619 06:55:50.788167 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:50.787169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34965 (local address 127.0.0.1:46320)
0619 06:55:50.787937 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:50.788051 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646}
W20260619 06:55:50.788354 4805 catalog_manager.cc:4584] TS 229e3e6391d74f73a9eabeaddc832bb7 (127.22.113.1:34965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6eae12c1847b440aa59a2edcd930222c: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34965: connect: Connection refused (error 111)
I20260619 06:55:50.794603 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5147
I20260619 06:55:50.801375 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 4786
I20260619 06:55:50.808391 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--webserver_interface=127.22.113.62
--webserver_port=35207
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:50.920208 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:50.920493 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:50.920537 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:50.924659 5313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:55:50.924728 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:50.924748 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:50.924765 5313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:55:50.924782 5313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:55:50.930188 5313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=35207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:50.931296 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:50.932346 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:50.941648 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.941589 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:50.941587 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:50.942065 5313 server_base.cc:1034] running on GCE node
I20260619 06:55:50.942775 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:50.944260 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:50.945453 5313 hybrid_clock.cc:648] HybridClock initialized: now 1781852150945416 us; error 59 us; skew 500 ppm
Jun 19 06:55:50 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852150, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:50.948859 5313 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:55:50.949982 5313 webserver.cc:460] Webserver started at http://127.22.113.62:35207/ using document root <none> and password file <none>
I20260619 06:55:50.950562 5313 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:50.950626 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:50.954785 5313 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:55:50.957129 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:50.958648 5313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:55:50.958802 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "094abe28bdc24678b8c74b074a1b01cd"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "b37773f704ae5b854858288d5a7bb0c2"
server_key_iv: "3cb1597b74715e95646e1d24b45e07b8"
server_key_version: "encryptionkey@0"
I20260619 06:55:50.959465 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:50.981529 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:50.984797 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:50.985026 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:50.992980 5313 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:38627
I20260619 06:55:50.993360 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:38627 every 8 connection(s)
I20260619 06:55:50.994217 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:55:50.995927 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5313
I20260619 06:55:50.996425 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:34965
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=32857
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:50.999850 5381 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:55:51.001928 5381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap starting.
I20260619 06:55:51.022955 5381 log.cc:824] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:51.035704 5381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:55:51.036211 5381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd: Bootstrap complete.
I20260619 06:55:51.041015 5381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:51.041509 5381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Initialized, Role: FOLLOWER
I20260619 06:55:51.042143 5381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:51.042307 5381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:55:51.042400 5381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:55:51.042528 5381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:55:51.044883 5381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:51.045267 5381 leader_election.cc:302] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094abe28bdc24678b8c74b074a1b01cd; no voters:
I20260619 06:55:51.045670 5381 leader_election.cc:288] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:55:51.045773 5385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:55:51.045992 5385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [term 3 LEADER]: Becoming Leader. State: Replica: 094abe28bdc24678b8c74b074a1b01cd, State: Running, Role: LEADER
I20260619 06:55:51.046309 5385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } }
I20260619 06:55:51.047096 5381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:55:51.047421 5387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094abe28bdc24678b8c74b074a1b01cd. Latest consensus state: current_term: 3 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:51.047538 5387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:51.047833 5386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "094abe28bdc24678b8c74b074a1b01cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094abe28bdc24678b8c74b074a1b01cd" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38627 } } }
I20260619 06:55:51.048038 5386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094abe28bdc24678b8c74b074a1b01cd [sys.catalog]: This master's current role is: LEADER
I20260619 06:55:51.049365 5389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:55:51.055228 5389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:55:51.055966 5389 catalog_manager.cc:1259] Loaded cluster ID: 96ba8d67202e48f0a4664b996a6e1894
I20260619 06:55:51.056031 5389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 06:55:51.057147 5389 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f27716dc9ad google::LogMessage::Fail() at ??:0
@ 0x7f27716df88a google::LogMessage::SendToLog() at ??:0
@ 0x7f27716dc4de google::LogMessage::Flush() at ??:0
@ 0x7f27716dda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f277c01bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f277c01cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f277c019579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f277c05d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f277392e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f277392ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f27739303f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f277391f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2774f636db start_thread at ??:0
@ 0x7f27701da61f clone at ??:0
W20260619 06:55:51.140108 5383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:51.140419 5383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:51.140468 5383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:51.144766 5383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:51.144845 5383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:51.144909 5383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:55:51.150642 5383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:34965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=32857
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:51.151898 5383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:51.152958 5383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:51.163447 5422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:51.163419 5425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:51.163420 5423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:51.164858 5383 server_base.cc:1034] running on GCE node
I20260619 06:55:51.165355 5383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:51.166352 5383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:51.167527 5383 hybrid_clock.cc:648] HybridClock initialized: now 1781852151167461 us; error 89 us; skew 500 ppm
Jun 19 06:55:51 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852151, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:51.170845 5383 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:55:51.171877 5383 webserver.cc:460] Webserver started at http://127.22.113.1:32857/ using document root <none> and password file <none>
I20260619 06:55:51.172596 5383 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:51.172693 5383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:51.177073 5383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:55:51.179872 5432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.181466 5383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:51.181633 5383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "229e3e6391d74f73a9eabeaddc832bb7"
format_stamp: "Formatted at 2026-06-19 06:55:48 on dist-test-slave-g8zg"
server_key: "765762330a28a59aff1abf1854799a46"
server_key_iv: "779d041c57b0ea40070a552297bc1adb"
server_key_version: "encryptionkey@0"
I20260619 06:55:51.182224 5383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:51.218494 5383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:51.222011 5383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:51.222239 5383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:51.222882 5383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:51.224231 5439 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:55:51.231204 5383 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:55:51.231395 5383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 06:55:51.231496 5383 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:55:51.233573 5383 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:55:51.233654 5383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 06:55:51.234036 5439 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap starting.
I20260619 06:55:51.245054 5383 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:34965
I20260619 06:55:51.246258 5383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:55:51.247299 5546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:34965 every 8 connection(s)
I20260619 06:55:51.254484 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5383
I20260619 06:55:51.254935 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:41127
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=33781
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:55:51.259812 5439 log.cc:824] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:55:51.270510 5439 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:55:51.271034 5439 tablet_bootstrap.cc:492] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Bootstrap complete.
I20260619 06:55:51.272526 5439 ts_tablet_manager.cc:1397] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.017s
I20260619 06:55:51.276180 5439 raft_consensus.cc:348] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:51.276752 5439 raft_consensus.cc:729] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e3e6391d74f73a9eabeaddc832bb7, State: Initialized, Role: FOLLOWER
I20260619 06:55:51.277349 5439 consensus_queue.cc:260] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:51.278548 5439 ts_tablet_manager.cc:1428] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 06:55:51.292918 5549 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:55:51.248436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:60971)
0619 06:55:51.248991 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.249045 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:51.249907 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:51.292000 (+ 42093us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":151,"threads_started":1}
W20260619 06:55:51.293710 5547 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:38627 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:38627: Client connection negotiation failed: client connection to 127.22.113.62:38627: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:55:51.294726 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.294350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:50823)
0619 06:55:51.294536 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.294643 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:55:51.295655 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.295370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:60831)
0619 06:55:51.295501 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.295559 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:55:51.295909 5547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:55:51.371685 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:51.371975 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:51.372020 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:51.376358 5550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:51.376437 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:51.376526 5550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:55:51.382479 5550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:41127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=33781
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:51.383749 5550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:51.384827 5550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:51.395560 5558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:51.395637 5550 server_base.cc:1034] running on GCE node
W20260619 06:55:51.395603 5557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:51.395560 5560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:51.396395 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:51.397243 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:51.398447 5550 hybrid_clock.cc:648] HybridClock initialized: now 1781852151398401 us; error 64 us; skew 500 ppm
Jun 19 06:55:51 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852151, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:51.401726 5550 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:55:51.402762 5550 webserver.cc:460] Webserver started at http://127.22.113.2:33781/ using document root <none> and password file <none>
I20260619 06:55:51.403432 5550 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:51.403533 5550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:51.407580 5550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:55:51.409950 5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.411478 5550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:55:51.411629 5550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "6ea9cc27f58aaf1662d28b55e0f5b8b4"
server_key_iv: "e20b6441605a71d5f77d23fff6f30646"
server_key_version: "encryptionkey@0"
I20260619 06:55:51.412299 5550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:51.426608 5550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:51.429656 5550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:51.429878 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:51.430488 5550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:51.431409 5550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:51.431476 5550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.431545 5550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:51.431592 5550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.441886 5550 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41127
I20260619 06:55:51.441962 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41127 every 8 connection(s)
I20260619 06:55:51.443007 5550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:55:51.447837 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.445875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:37129)
0619 06:55:51.446497 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.447008 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":158,"threads_started":1}
W20260619 06:55:51.448421 5681 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:38627 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:38627: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
I20260619 06:55:51.449283 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.449021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:33353)
0619 06:55:51.449158 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.449215 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:55:51.450109 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.449770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:37499)
0619 06:55:51.449997 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.450046 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260619 06:55:51.450305 5681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:55:51.452265 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5550
I20260619 06:55:51.452773 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:33335
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:38627
--webserver_port=35849
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:55:51.577775 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:55:51.578084 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:55:51.578136 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:55:51.582352 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:55:51.582433 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:55:51.582499 5684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:55:51.588074 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:33335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=35849
--tserver_master_addrs=127.22.113.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:55:51.589177 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:55:51.590263 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:55:51.601238 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:55:51.601266 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:51.601333 5684 server_base.cc:1034] running on GCE node
W20260619 06:55:51.601253 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:55:51.602029 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:55:51.602797 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:55:51.604005 5684 hybrid_clock.cc:648] HybridClock initialized: now 1781852151603951 us; error 71 us; skew 500 ppm
Jun 19 06:55:51 dist-test-slave-g8zg krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852151, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:55:51.607029 5684 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:55:51.608029 5684 webserver.cc:460] Webserver started at http://127.22.113.3:35849/ using document root <none> and password file <none>
I20260619 06:55:51.608594 5684 fs_manager.cc:359] Metadata directory not provided
I20260619 06:55:51.608656 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:55:51.612866 5684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:55:51.615170 5699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.616746 5684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:55:51.616923 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "015c641dcebb429ebe13198d7772e5a6"
format_stamp: "Formatted at 2026-06-19 06:55:49 on dist-test-slave-g8zg"
server_key: "b5a8911bc95afd72ea3f5a355d85c437"
server_key_iv: "3d0b0ea01a5273942ab0f921ee5c31e9"
server_key_version: "encryptionkey@0"
I20260619 06:55:51.617492 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:55:51.629073 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:55:51.632509 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:55:51.632733 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:55:51.633360 5684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:55:51.634239 5684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:55:51.634316 5684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.634389 5684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:55:51.634439 5684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:55:51.644927 5684 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:33335
I20260619 06:55:51.644997 5812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:33335 every 8 connection(s)
I20260619 06:55:51.646131 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:55:51.650949 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5684
I20260619 06:55:51.651204 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.648944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:45131)
0619 06:55:51.649576 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.649720 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":176,"threads_started":1}
W20260619 06:55:51.651895 5813 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:38627 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:38627: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
I20260619 06:55:51.652626 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.652392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:34583)
0619 06:55:51.652519 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.652568 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:55:51.653355 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:51.653161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:59527)
0619 06:55:51.653267 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:51.653310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:55:51.653546 5813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:55:52.297308 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.296655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:46351)
0619 06:55:52.297044 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:52.297150 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260619 06:55:52.451674 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.450986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:60039)
0619 06:55:52.451438 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:52.451520 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":98,"threads_started":1}
I20260619 06:55:52.592303 5819 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:52.592507 5819 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:52.593652 5819 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127), 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
Jun 19 06:55:52 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852151, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.2@KRBTEST.COM
Jun 19 06:55:52 dist-test-slave-g8zg krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852151, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.3@KRBTEST.COM
I20260619 06:55:52.610263 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.594030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41127 (local address 127.22.113.1:56275)
0619 06:55:52.594636 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:52.594651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:52.594805 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:52.596370 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:52.596376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:52.597079 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:52.597650 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:52.597669 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.599134 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.599138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:52.600319 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:52.600329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.601282 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.601286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:52.601576 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:52.602379 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:52.602397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:52.604273 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:52.607080 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:52.607083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:52.607085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:52.607445 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:52.608129 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:52.608132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:52.608134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:52.608220 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:52.608921 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:52.608928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:52.609345 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 06:55:52.609591 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 06:55:52.610058 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":400,"spinlock_wait_cycles":47104,"thread_start_us":107,"threads_started":1}
I20260619 06:55:52.610306 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.594196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:37553 (local address 127.22.113.3:33335)
0619 06:55:52.594733 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:52.594746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:52.594784 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:55:52.595670 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:52.595677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:52.596101 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:52.596453 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:52.597841 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.598926 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.600467 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.601144 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.601338 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:52.604784 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:52.604791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:52.604809 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:52.604850 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:52.606797 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:52.607610 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:52.607616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:52.607621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:52.607792 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:52.608188 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:52.608190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:52.608192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:52.608780 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:52.608929 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:52.609769 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 06:55:52.610102 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":118,"threads_started":1}
I20260619 06:55:52.610306 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.594163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:56275 (local address 127.22.113.2:41127)
0619 06:55:52.594482 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 06:55:52.594508 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 06:55:52.594757 (+ 249us) server_negotiation.cc:407] Connection header received
0619 06:55:52.595631 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:55:52.595638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:55:52.596000 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:55:52.596324 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:55:52.597826 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.599004 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.600502 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.601149 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.601329 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:52.604858 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:55:52.604863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:55:52.604878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:55:52.604911 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:55:52.606963 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:52.607921 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:52.607925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:52.607928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:52.608027 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:55:52.608333 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:55:52.608336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:55:52.608338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:55:52.608780 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:55:52.608926 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:55:52.609769 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 06:55:52.610100 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":124,"threads_started":1}
I20260619 06:55:52.610263 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.593939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:33335 (local address 127.22.113.1:37553)
0619 06:55:52.594420 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:52.594439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:55:52.594580 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:55:52.596388 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:55:52.596392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:55:52.597080 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:55:52.597648 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:52.597669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.599081 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.599085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:52.600324 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:55:52.600337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:55:52.601279 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:55:52.601283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:55:52.601578 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:55:52.602526 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:55:52.602539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:55:52.604548 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:55:52.606926 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:52.606933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:52.606949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:52.607445 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:52.607895 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:55:52.607897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:55:52.607899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:55:52.608037 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:55:52.608921 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:55:52.608928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:55:52.609344 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 06:55:52.609591 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 06:55:52.610004 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":278,"spinlock_wait_cycles":22400}
W20260619 06:55:52.613541 5436 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
W20260619 06:55:52.613740 5434 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:52.613868 5434 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7; no voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741
I20260619 06:55:52.614187 5819 raft_consensus.cc:2726] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:55:52.654763 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:52.654253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:58925)
0619 06:55:52.654549 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:52.654623 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260619 06:55:53.299046 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:53.298493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:50201)
0619 06:55:53.298847 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:53.298924 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260619 06:55:53.453312 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:53.452777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:57459)
0619 06:55:53.453116 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:53.453200 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260619 06:55:53.656092 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:53.655653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:55919)
0619 06:55:53.655900 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:53.655969 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260619 06:55:54.300733 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:54.300135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:40323)
0619 06:55:54.300529 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:54.300613 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260619 06:55:54.454975 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:54.454397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:39793)
0619 06:55:54.454737 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:54.454809 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260619 06:55:54.605563 5829 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:54.605760 5829 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:54.606336 5829 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127), 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
W20260619 06:55:54.607535 5434 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
W20260619 06:55:54.607736 5436 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:54.607900 5436 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7; no voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741
I20260619 06:55:54.608129 5829 raft_consensus.cc:2726] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:55:54.657774 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:54.657155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:57913)
0619 06:55:54.657583 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:54.657662 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260619 06:55:55.302225 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:55.301673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:54239)
0619 06:55:55.302043 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:55.302118 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260619 06:55:55.456562 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:55.456021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:33517)
0619 06:55:55.456385 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:55.456450 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260619 06:55:55.659533 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:55.658756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:54707)
0619 06:55:55.659148 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:55.659228 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":90,"threads_started":1}
I20260619 06:55:56.304278 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:56.303600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:39103)
0619 06:55:56.303995 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:56.304117 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":96,"threads_started":1}
I20260619 06:55:56.458266 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:56.457705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:58357)
0619 06:55:56.458059 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:56.458140 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260619 06:55:56.473582 5836 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:55:56.473766 5836 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:55:56.474193 5836 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127), 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
W20260619 06:55:56.475179 5436 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
W20260619 06:55:56.475353 5434 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
I20260619 06:55:56.475400 5436 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7; no voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741
I20260619 06:55:56.475574 5836 raft_consensus.cc:2726] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:55:56.661394 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:56.660616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:37881)
0619 06:55:56.661150 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:56.661260 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":96,"threads_started":1}
I20260619 06:55:57.305949 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:57.305328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:60897)
0619 06:55:57.305716 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:57.305836 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260619 06:55:57.459827 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:57.459297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:40703)
0619 06:55:57.459637 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:57.459707 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260619 06:55:57.663033 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:57.662434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:45723)
0619 06:55:57.662804 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:57.662916 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260619 06:55:58.307561 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:58.306930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:34541)
0619 06:55:58.307371 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:58.307455 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260619 06:55:58.461315 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:58.460693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:47599)
0619 06:55:58.461105 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:58.461189 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":91,"threads_started":1}
I20260619 06:55:58.664387 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:58.663828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:41609)
0619 06:55:58.664196 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:58.664262 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":80,"threads_started":1}
I20260619 06:55:59.309151 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:59.308584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:41667)
0619 06:55:59.308922 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:59.309009 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260619 06:55:59.462895 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:59.462305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:39467)
0619 06:55:59.462684 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:59.462781 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260619 06:55:59.665951 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:55:59.665352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:40981)
0619 06:55:59.665733 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 06:55:59.665818 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":82,"threads_started":1}
I20260619 06:56:00.310551 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:00.309978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:51775)
0619 06:56:00.310339 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:00.310430 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":97,"threads_started":1}
I20260619 06:56:00.464416 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:00.463923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:54479)
0619 06:56:00.464219 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:00.464289 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260619 06:56:00.667513 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:00.666913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:33905)
0619 06:56:00.667320 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:00.667405 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260619 06:56:01.312492 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:01.311923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:54275)
0619 06:56:01.312298 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:01.312384 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260619 06:56:01.427704 5851 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:01.427855 5851 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:56:01.428382 5851 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127), 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
W20260619 06:56:01.429409 5434 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
W20260619 06:56:01.429612 5436 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
I20260619 06:56:01.429694 5436 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7; no voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741
I20260619 06:56:01.429874 5851 raft_consensus.cc:2726] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:01.466130 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:01.465530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:45103)
0619 06:56:01.465901 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:01.465985 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
I20260619 06:56:01.669081 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:01.668450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:57587)
0619 06:56:01.668888 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:01.668972 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":123,"threads_started":1}
I20260619 06:56:02.314103 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:02.313583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:39733)
0619 06:56:02.313898 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:02.313971 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260619 06:56:02.467873 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:02.467215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:34883)
0619 06:56:02.467663 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:02.467752 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":105,"threads_started":1}
I20260619 06:56:02.670797 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:02.670211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:40891)
0619 06:56:02.670584 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:02.670669 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260619 06:56:03.316080 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:03.315315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:43077)
0619 06:56:03.315813 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:03.315912 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":132,"threads_started":1}
I20260619 06:56:03.469642 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:03.469067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:55215)
0619 06:56:03.469398 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:03.469474 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260619 06:56:03.672431 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:03.671867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:57899)
0619 06:56:03.672215 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:03.672298 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260619 06:56:03.775503 5860 raft_consensus.cc:482] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:03.775648 5860 raft_consensus.cc:504] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9f39ce2d9c4c4e9dd0c1551376b741" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41127 } } peers { permanent_uuid: "015c641dcebb429ebe13198d7772e5a6" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 33335 } } peers { permanent_uuid: "229e3e6391d74f73a9eabeaddc832bb7" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34965 } }
I20260619 06:56:03.776059 5860 leader_election.cc:288] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127), 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335)
W20260619 06:56:03.777199 5436 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 015c641dcebb429ebe13198d7772e5a6 (127.22.113.3:33335): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
W20260619 06:56:03.777376 5434 leader_election.cc:341] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d9f39ce2d9c4c4e9dd0c1551376b741 (127.22.113.2:41127): Not found: Tablet not found: 6eae12c1847b440aa59a2edcd930222c
I20260619 06:56:03.777451 5434 leader_election.cc:302] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 229e3e6391d74f73a9eabeaddc832bb7; no voters: 015c641dcebb429ebe13198d7772e5a6, 3d9f39ce2d9c4c4e9dd0c1551376b741
I20260619 06:56:03.777609 5860 raft_consensus.cc:2726] T 6eae12c1847b440aa59a2edcd930222c P 229e3e6391d74f73a9eabeaddc832bb7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:04.317896 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:04.317314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:32851)
0619 06:56:04.317678 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:04.317749 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260619 06:56:04.471225 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:04.470602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:43787)
0619 06:56:04.470991 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:04.471090 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260619 06:56:04.673969 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:04.673370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:36925)
0619 06:56:04.673749 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:04.673830 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260619 06:56:05.319463 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:05.318880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:50027)
0619 06:56:05.319218 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:05.319329 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":140,"threads_started":1}
I20260619 06:56:05.472908 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:05.472345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:47157)
0619 06:56:05.472676 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:05.472743 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260619 06:56:05.675643 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:05.675002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.3:55221)
0619 06:56:05.675447 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:05.675515 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":135,"threads_started":1}
I20260619 06:56:06.321142 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:06.320592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.1:48981)
0619 06:56:06.320907 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:06.321007 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260619 06:56:06.474843 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:06.474261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:38627 (local address 127.22.113.2:45617)
0619 06:56:06.474625 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:06.474735 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260619 06:56:06.654841 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5383
I20260619 06:56:06.661582 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5550
I20260619 06:56:06.668414 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5684
2026-06-19T06:56:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19473 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:56:06 dist-test-slave-g8zg krb5kdc[5871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:56:06 dist-test-slave-g8zg krb5kdc[5871](info): set up 2 sockets
Jun 19 06:56:06 dist-test-slave-g8zg krb5kdc[5871](info): commencing operation
krb5kdc: starting...
W20260619 06:56:07.719645 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:56:07 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:56:07.863466 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:07.989233 5884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:07.989642 5884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:07.989728 5884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:07.994410 5884 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:56:07.994526 5884 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:56:07.994563 5884 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:56:07.994588 5884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:56:07.994637 5884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:07.994683 5884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:07.994741 5884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:56:07.994791 5884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:56:08.001004 5884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:08.002610 5884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:08.004002 5884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:08.014200 5889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:08.014200 5890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.014671 5884 server_base.cc:1034] running on GCE node
W20260619 06:56:08.014335 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.015491 5884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:08.016908 5884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:08.018083 5884 hybrid_clock.cc:648] HybridClock initialized: now 1781852168018042 us; error 58 us; skew 500 ppm
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:08.021569 5884 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:56:08.022871 5884 webserver.cc:460] Webserver started at http://127.22.113.62:35607/ using document root <none> and password file <none>
I20260619 06:56:08.023648 5884 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:08.023746 5884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:08.024014 5884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:56:08.026476 5884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "4de7f71695ff4b4d95b5f194bf27bcb4"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "2c8df6460abd3d286845635618a2841e"
server_key_iv: "e1fb5eda1121c0ef66086c377345699f"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.027148 5884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "4de7f71695ff4b4d95b5f194bf27bcb4"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "2c8df6460abd3d286845635618a2841e"
server_key_iv: "e1fb5eda1121c0ef66086c377345699f"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.031818 5884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:56:08.034718 5899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.036257 5884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:56:08.036417 5884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "4de7f71695ff4b4d95b5f194bf27bcb4"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "2c8df6460abd3d286845635618a2841e"
server_key_iv: "e1fb5eda1121c0ef66086c377345699f"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.036547 5884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:08.053985 5884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:08.057686 5884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:08.057926 5884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:08.067324 5884 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40579
I20260619 06:56:08.067730 5951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40579 every 8 connection(s)
I20260619 06:56:08.068542 5884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:56:08.071318 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5884
I20260619 06:56:08.071486 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:56:08.071832 22980 external_mini_cluster.cc:1428] Setting key 06a7dc6c20971702426f497c3288ae34
I20260619 06:56:08.071828 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:08.079128 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap starting.
I20260619 06:56:08.082031 5952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Neither blocks nor log segments found. Creating new log.
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:56:08.082827 5952 log.cc:824] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:08.084770 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: No bootstrap required, opened a new log
I20260619 06:56:08.088503 5952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:08.088758 5952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:08.088797 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.073485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.22.113.62:40579)
0619 06:56:08.073990 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.074008 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.074052 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:56:08.075311 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.075337 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.075743 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.076081 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:08.077063 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.078127 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.079385 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.079985 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.080140 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.083444 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:08.083449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:08.083466 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:08.083498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:08.085481 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.085957 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.085963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.085969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.086084 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.086374 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.086378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.086379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.086851 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:08.087069 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.087401 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.087772 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260619 06:56:08.088841 5952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Initialized, Role: FOLLOWER
I20260619 06:56:08.089413 5952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:08.089574 5952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:56:08.089645 5952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:56:08.089742 5952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:08.090677 5952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:08.091035 5952 leader_election.cc:302] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de7f71695ff4b4d95b5f194bf27bcb4; no voters:
I20260619 06:56:08.091454 5952 leader_election.cc:288] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:56:08.091786 5957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:56:08.092008 5957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 LEADER]: Becoming Leader. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Running, Role: LEADER
I20260619 06:56:08.092298 5957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:08.092773 5952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:56:08.094265 5959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de7f71695ff4b4d95b5f194bf27bcb4. Latest consensus state: current_term: 1 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:08.094326 5958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:08.094635 5958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:08.094993 5959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:08.095072 5970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:56:08.098125 5970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:56:08.103753 5970 catalog_manager.cc:1347] Generated new cluster ID: aece56151e8a4ad6a95e92a6a0c45cd8
I20260619 06:56:08.103832 5970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:56:08.116688 5970 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:56:08.117805 5970 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:56:08.128509 5970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Generated new TSK 0
I20260619 06:56:08.129678 5970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:56:08.166778 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:08.282930 5978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:08.283203 5978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:08.283305 5978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:08.287281 5978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:08.287343 5978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:08.287402 5978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:56:08.292691 5978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:08.293860 5978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:08.294958 5978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:08.305349 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:08.305356 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:08.305371 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.305524 5978 server_base.cc:1034] running on GCE node
I20260619 06:56:08.306119 5978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:08.306793 5978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:08.307981 5978 hybrid_clock.cc:648] HybridClock initialized: now 1781852168307938 us; error 57 us; skew 500 ppm
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:08.311017 5978 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:56:08.311983 5978 webserver.cc:460] Webserver started at http://127.22.113.1:39995/ using document root <none> and password file <none>
I20260619 06:56:08.312542 5978 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:08.312603 5978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:08.312770 5978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:56:08.314802 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "cc5949f652fc46df9facc33661d6d4ec"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "8d129d9f21badcff2176bd024d68435c"
server_key_iv: "812eb847736fe32fc093305d4acfac4b"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.315390 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "cc5949f652fc46df9facc33661d6d4ec"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "8d129d9f21badcff2176bd024d68435c"
server_key_iv: "812eb847736fe32fc093305d4acfac4b"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.319406 5978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 06:56:08.321950 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.323599 5978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:56:08.323743 5978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "cc5949f652fc46df9facc33661d6d4ec"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "8d129d9f21badcff2176bd024d68435c"
server_key_iv: "812eb847736fe32fc093305d4acfac4b"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.323904 5978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:08.350863 5978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:08.354991 5978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:08.355333 5978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:08.356132 5978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:08.357214 5978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:08.357280 5978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.357366 5978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:08.357405 5978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.369302 5978 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46851
I20260619 06:56:08.369381 6106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46851 every 8 connection(s)
I20260619 06:56:08.370510 5978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:56:08.375126 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 5978
I20260619 06:56:08.375365 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:56:08.375818 22980 external_mini_cluster.cc:1428] Setting key a738b7b50b90f6d50b5c972867426976
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:56:08.389564 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.372771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44153 (local address 127.22.113.62:40579)
0619 06:56:08.373036 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.373043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.374165 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 06:56:08.375725 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.375728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.375808 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.375963 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:08.377477 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.378122 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.379945 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.380392 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.380519 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.384497 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:08.384504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:08.384510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:08.384553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:08.386650 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.387481 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.387485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.387487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.387617 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.388084 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.388087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.388089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.388431 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:08.388546 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.389223 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.389378 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:56:08.390630 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.373338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:44153)
0619 06:56:08.373966 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.374019 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.375475 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.376160 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.376167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.376631 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:08.377218 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.377231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.378277 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.378294 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.379781 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.379791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.380753 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.380759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.381080 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.381905 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:08.381932 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:08.384262 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:08.386801 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.386806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.386875 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.387368 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.387720 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.387724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.387727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.387959 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.388583 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:08.388590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:08.388955 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.389166 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.389509 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:56:08.392517 6107 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:08.392868 6107 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:08.393621 6107 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:08.395813 5916 ts_manager.cc:194] Registered new tserver with Master: cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851)
I20260619 06:56:08.397728 5916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:44153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:56:08.410902 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:08.549230 6112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:08.549847 6112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:08.549937 6112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:08.558523 6112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:08.558658 6112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:08.558779 6112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:56:08.571329 6112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:08.573385 6112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:08.575273 6112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:08.589684 6117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:08.589888 6118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.590060 6112 server_base.cc:1034] running on GCE node
W20260619 06:56:08.589888 6120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.590698 6112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:08.591545 6112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:08.592774 6112 hybrid_clock.cc:648] HybridClock initialized: now 1781852168592743 us; error 43 us; skew 500 ppm
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:08.596241 6112 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:56:08.597361 6112 webserver.cc:460] Webserver started at http://127.22.113.2:42795/ using document root <none> and password file <none>
I20260619 06:56:08.598088 6112 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:08.598250 6112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:08.598546 6112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:56:08.601178 6112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "67c03fde360b4ba4a226fd763aef14d3"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "3639b2587d34829f3c9376701bec1898"
server_key_iv: "c5e08146540d43f7ccff0b553a0f0466"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.601854 6112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "67c03fde360b4ba4a226fd763aef14d3"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "3639b2587d34829f3c9376701bec1898"
server_key_iv: "c5e08146540d43f7ccff0b553a0f0466"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.606660 6112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:56:08.609778 6127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.611479 6112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:56:08.611632 6112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "67c03fde360b4ba4a226fd763aef14d3"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "3639b2587d34829f3c9376701bec1898"
server_key_iv: "c5e08146540d43f7ccff0b553a0f0466"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.611783 6112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:08.648694 6112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:08.652860 6112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:08.653102 6112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:08.653827 6112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:08.654819 6112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:08.654893 6112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.654973 6112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:08.655015 6112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.666464 6112 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:32783
I20260619 06:56:08.666502 6240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:32783 every 8 connection(s)
I20260619 06:56:08.667610 6112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:56:08.669143 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6112
I20260619 06:56:08.669274 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:56:08.669607 22980 external_mini_cluster.cc:1428] Setting key 1c139872571ea8b516b95c5a31c632b2
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:56:08.684875 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.670247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:57027 (local address 127.22.113.62:40579)
0619 06:56:08.670433 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.670437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.671320 (+ 883us) server_negotiation.cc:407] Connection header received
0619 06:56:08.673375 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.673378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.673450 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.673538 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:08.674874 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.675623 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.677458 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.677684 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.677771 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.680909 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:08.680913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:08.680917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:08.680951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:08.682511 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.683153 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.683158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.683160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.683287 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.683710 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.683713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.683715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.683906 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:08.684017 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.684556 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.684690 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:56:08.685873 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.670483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:57027)
0619 06:56:08.671112 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.671177 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.673099 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.673788 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.673799 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.674210 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:08.674711 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.674724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.675759 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.675764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.677320 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.677333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.677858 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.677866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.678105 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.678823 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:08.678849 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:08.680718 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:08.682651 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.682658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.682675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.683041 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.683389 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.683393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.683396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.683601 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.684015 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:08.684022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:08.684312 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.684517 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.684894 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":124,"threads_started":1}
I20260619 06:56:08.687124 6241 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:08.687439 6241 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:08.687990 6241 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:08.689132 5916 ts_manager.cc:194] Registered new tserver with Master: 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
I20260619 06:56:08.689777 5916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:57027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:56:08.700403 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:08.817762 6246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:08.818011 6246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:08.818058 6246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:08.822000 6246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:08.822059 6246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:08.822116 6246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:56:08.827672 6246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:08.828649 6246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:08.829610 6246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:08.840246 6254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:08.840246 6251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.840683 6246 server_base.cc:1034] running on GCE node
W20260619 06:56:08.840591 6252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:08.841285 6246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:08.841964 6246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:08.843223 6246 hybrid_clock.cc:648] HybridClock initialized: now 1781852168843205 us; error 45 us; skew 500 ppm
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:08.846475 6246 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:56:08.847718 6246 webserver.cc:460] Webserver started at http://127.22.113.3:36865/ using document root <none> and password file <none>
I20260619 06:56:08.848413 6246 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:08.848484 6246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:08.848744 6246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:56:08.851207 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "6f966db3513d4d5fb735f1e598d20c9e"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "a6769eb2b93e3498dfba2b2377a828b6"
server_key_iv: "6c766a93719a156d095d353ff3bbba0e"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.851933 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "6f966db3513d4d5fb735f1e598d20c9e"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "a6769eb2b93e3498dfba2b2377a828b6"
server_key_iv: "6c766a93719a156d095d353ff3bbba0e"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.856906 6246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:56:08.860071 6261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.861889 6246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:56:08.862039 6246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6f966db3513d4d5fb735f1e598d20c9e"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "a6769eb2b93e3498dfba2b2377a828b6"
server_key_iv: "6c766a93719a156d095d353ff3bbba0e"
server_key_version: "encryptionkey@0"
I20260619 06:56:08.862180 6246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:08.875661 6246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:08.879649 6246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:08.879892 6246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:08.880654 6246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:08.881812 6246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:08.881870 6246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.881915 6246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:08.881932 6246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:08.894805 6246 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38605
I20260619 06:56:08.895001 6374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38605 every 8 connection(s)
I20260619 06:56:08.896010 6246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:56:08.897676 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6246
I20260619 06:56:08.897825 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:56:08.898238 22980 external_mini_cluster.cc:1428] Setting key 8c5cb49893141eb2f59001095d82029c
Jun 19 06:56:08 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:56:08.910956 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.898383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:38875 (local address 127.22.113.62:40579)
0619 06:56:08.898563 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.898566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.899769 (+ 1203us) server_negotiation.cc:407] Connection header received
0619 06:56:08.901466 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.901469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.901525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.901633 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:08.902823 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.903430 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.904553 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.904775 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.904887 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.907638 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:08.907641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:08.907643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:08.907664 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:08.908990 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.909483 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.909487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.909489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.909573 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.909910 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.909915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.909917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.910090 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:08.910200 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.910666 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.910781 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:56:08.911899 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.898951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:38875)
0619 06:56:08.899601 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.899667 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.901256 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.901815 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.901822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.902197 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:08.902673 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.902686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.903550 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.903554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.904426 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.904435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.904925 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.904929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.905100 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.905745 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:08.905761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:08.907476 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:08.909108 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.909113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.909126 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.909379 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.909667 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:08.909671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:08.909673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:08.909814 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:08.910195 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:08.910201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:08.910441 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.910626 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.910935 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":172,"threads_started":1}
I20260619 06:56:08.913035 6375 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:08.913407 6375 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:08.914108 6375 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:08.915385 5915 ts_manager.cc:194] Registered new tserver with Master: 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605)
I20260619 06:56:08.916183 5915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:38875
I20260619 06:56:08.925288 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:56:08.936159 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.927441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.22.113.62:40579)
0619 06:56:08.927594 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.927598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.928095 (+ 497us) server_negotiation.cc:407] Connection header received
0619 06:56:08.928287 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.928295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.928370 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.928514 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:08.929256 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.929914 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.930799 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.931042 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.931135 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.932634 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:08.932638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:08.932640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:08.932665 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:08.934362 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.934818 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.934821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.934823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.934907 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:08.935177 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:08.935179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:08.935180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:08.935494 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:08.935652 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.935909 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.936019 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:56:08.939426 5915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:56:08.942739 5915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:56:08.958992 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.952619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.0.0.1:50142)
0619 06:56:08.953664 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.953722 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.953868 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.954621 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.954628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.954662 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:08.955000 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.955007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956233 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.956238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.957545 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.957554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.958393 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.958396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.958522 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.958563 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.958701 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.958846 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":176,"spinlock_wait_cycles":19456,"thread_start_us":145,"threads_started":1}
I20260619 06:56:08.958998 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.953144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.22.113.1:46851)
0619 06:56:08.953785 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.953793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.953823 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:56:08.954078 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.954084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.954306 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.954450 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:08.955353 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956107 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.957671 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.958263 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.958395 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.958547 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.958732 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.958833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":103,"threads_started":1}
I20260619 06:56:08.959951 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.952592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.22.113.3:38605)
0619 06:56:08.953395 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.953410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.954010 (+ 600us) server_negotiation.cc:407] Connection header received
0619 06:56:08.954082 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.954087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.954301 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.954450 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:08.955154 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956507 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.958422 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.959299 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.959405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.959586 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.959644 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.959751 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":228,"threads_started":1}
I20260619 06:56:08.960407 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.953314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:32783 (local address 127.0.0.1:35680)
0619 06:56:08.953992 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.954008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.954110 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.954798 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.954801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.954820 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:08.955015 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.955022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956675 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.956678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.958298 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.958305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.960004 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.960010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.960137 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.960163 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.960219 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.960298 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":194,"thread_start_us":143,"threads_started":1}
I20260619 06:56:08.960788 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.952480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38605 (local address 127.0.0.1:35204)
0619 06:56:08.953664 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:08.953722 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:08.953868 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:08.954621 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:08.954628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:08.954663 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:08.954988 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.954998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956675 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.956678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.958273 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:08.958284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.959419 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.959424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:08.959520 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.959543 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:56:08.959612 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:56:08.959693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":195,"spinlock_wait_cycles":20736,"thread_start_us":106,"threads_started":1}
I20260619 06:56:08.961185 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:08.953815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.22.113.2:32783)
0619 06:56:08.954362 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:08.954373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:08.954395 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:56:08.954482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:08.954487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:08.954656 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:08.954827 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:08.955174 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.956513 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.958421 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:08.959856 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:08.960730 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:08.960837 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:08.960904 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:56:08.961001 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":131,"threads_started":1}
I20260619 06:56:08.962582 6041 tablet_service.cc:1467] Processing CreateTablet for tablet 830d467569ee4e22b2898aadec309b76 (DEFAULT_TABLE table=test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:08.963828 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d467569ee4e22b2898aadec309b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:08.963783 6175 tablet_service.cc:1467] Processing CreateTablet for tablet 830d467569ee4e22b2898aadec309b76 (DEFAULT_TABLE table=test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:08.964740 6175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d467569ee4e22b2898aadec309b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:08.969674 6309 tablet_service.cc:1467] Processing CreateTablet for tablet 830d467569ee4e22b2898aadec309b76 (DEFAULT_TABLE table=test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:08.970772 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d467569ee4e22b2898aadec309b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:08.971575 6392 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Bootstrap starting.
I20260619 06:56:08.974431 6392 tablet_bootstrap.cc:654] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:08.975082 6393 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Bootstrap starting.
I20260619 06:56:08.975505 6392 log.cc:824] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:08.977869 6393 tablet_bootstrap.cc:654] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:08.978029 6392 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: No bootstrap required, opened a new log
I20260619 06:56:08.978253 6392 ts_tablet_manager.cc:1397] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 06:56:08.978749 6393 log.cc:824] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:08.980687 6393 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: No bootstrap required, opened a new log
I20260619 06:56:08.980907 6393 ts_tablet_manager.cc:1397] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:56:08.982755 6392 raft_consensus.cc:348] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.983029 6392 raft_consensus.cc:374] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:08.983110 6392 raft_consensus.cc:729] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5949f652fc46df9facc33661d6d4ec, State: Initialized, Role: FOLLOWER
I20260619 06:56:08.983657 6392 consensus_queue.cc:260] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.984493 6107 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
I20260619 06:56:08.984513 6393 raft_consensus.cc:348] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.984773 6392 ts_tablet_manager.cc:1428] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 06:56:08.984835 6393 raft_consensus.cc:374] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:08.984894 6393 raft_consensus.cc:729] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c03fde360b4ba4a226fd763aef14d3, State: Initialized, Role: FOLLOWER
I20260619 06:56:08.985030 6395 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Bootstrap starting.
I20260619 06:56:08.985371 6393 consensus_queue.cc:260] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.986016 6241 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
I20260619 06:56:08.986411 6393 ts_tablet_manager.cc:1428] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:56:08.987813 6395 tablet_bootstrap.cc:654] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:08.989387 6395 log.cc:824] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:08.991984 6395 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: No bootstrap required, opened a new log
I20260619 06:56:08.992215 6395 ts_tablet_manager.cc:1397] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260619 06:56:08.995731 6395 raft_consensus.cc:348] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.995995 6395 raft_consensus.cc:374] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:08.996030 6395 raft_consensus.cc:729] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f966db3513d4d5fb735f1e598d20c9e, State: Initialized, Role: FOLLOWER
I20260619 06:56:08.996497 6395 consensus_queue.cc:260] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:08.997260 6375 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
I20260619 06:56:08.997597 6395 ts_tablet_manager.cc:1428] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 06:56:09.122258 6108 tablet.cc:2369] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:56:09.147217 6376 tablet.cc:2369] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:56:09.163225 6400 raft_consensus.cc:482] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:09.163563 6400 raft_consensus.cc:504] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:09.164860 6400 leader_election.cc:288] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
W20260619 06:56:09.168956 6242 tablet.cc:2369] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:56:09.170581 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.165943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:42089 (local address 127.22.113.2:32783)
0619 06:56:09.166075 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:09.166079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:09.166362 (+ 283us) server_negotiation.cc:407] Connection header received
0619 06:56:09.166461 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:09.166464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:09.166556 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:09.166679 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:09.166943 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.167693 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.169356 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.169849 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.169998 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:09.170061 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:09.170347 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 06:56:09.170428 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:56:09.170625 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.165524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:33399 (local address 127.22.113.1:46851)
0619 06:56:09.165749 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:09.165755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:09.165780 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:56:09.165981 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:09.165985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:09.166056 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:09.166161 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:09.166565 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.167431 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.169272 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.169877 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.170022 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:09.170097 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:09.170382 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:56:09.170473 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:56:09.170660 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.165731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:32783 (local address 127.22.113.3:42089)
0619 06:56:09.166233 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:09.166249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:09.166361 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:09.166712 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:09.166714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:09.166730 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:09.166820 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:09.166825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.167818 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.167822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:09.169225 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:09.169234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.170000 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.170004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:09.170135 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:09.170225 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:56:09.170327 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:56:09.170412 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":53,"thread_start_us":200,"threads_started":1}
I20260619 06:56:09.170665 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.165405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.22.113.3:33399)
0619 06:56:09.165573 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:09.165595 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:09.165791 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:09.166215 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:09.166219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:09.166241 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:09.166417 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:09.166427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.167596 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.167599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:09.169100 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:09.169111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.170017 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.170021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:09.170249 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:09.170270 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:56:09.170364 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:56:09.170464 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":93}
I20260619 06:56:09.171458 6195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d467569ee4e22b2898aadec309b76" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c03fde360b4ba4a226fd763aef14d3" is_pre_election: true
I20260619 06:56:09.171746 6195 raft_consensus.cc:2454] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 0.
I20260619 06:56:09.171775 6061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d467569ee4e22b2898aadec309b76" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5949f652fc46df9facc33661d6d4ec" is_pre_election: true
I20260619 06:56:09.172150 6061 raft_consensus.cc:2454] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 0.
I20260619 06:56:09.172283 6265 leader_election.cc:302] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e; no voters:
I20260619 06:56:09.172634 6400 raft_consensus.cc:2781] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:56:09.172744 6400 raft_consensus.cc:482] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:56:09.172782 6400 raft_consensus.cc:3037] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:09.174057 6400 raft_consensus.cc:504] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:09.174484 6400 leader_election.cc:288] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 election: Requested vote from peers cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
I20260619 06:56:09.175022 6061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d467569ee4e22b2898aadec309b76" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5949f652fc46df9facc33661d6d4ec"
I20260619 06:56:09.175074 6195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d467569ee4e22b2898aadec309b76" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c03fde360b4ba4a226fd763aef14d3"
I20260619 06:56:09.175202 6061 raft_consensus.cc:3037] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:09.175292 6195 raft_consensus.cc:3037] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:09.176435 6061 raft_consensus.cc:2454] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 1.
I20260619 06:56:09.176669 6195 raft_consensus.cc:2454] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 1.
I20260619 06:56:09.176865 6265 leader_election.cc:302] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f966db3513d4d5fb735f1e598d20c9e, cc5949f652fc46df9facc33661d6d4ec; no voters:
I20260619 06:56:09.177078 6400 raft_consensus.cc:2781] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:56:09.177345 6400 raft_consensus.cc:686] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 1 LEADER]: Becoming Leader. State: Replica: 6f966db3513d4d5fb735f1e598d20c9e, State: Running, Role: LEADER
I20260619 06:56:09.177713 6400 consensus_queue.cc:237] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:09.181543 5914 catalog_manager.cc:5495] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3). New cstate: current_term: 1 leader_uuid: "6f966db3513d4d5fb735f1e598d20c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } health_report { overall_health: UNKNOWN } } }
I20260619 06:56:09.197922 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.194008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.22.113.3:38605)
0619 06:56:09.194176 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:09.194180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:09.194196 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:56:09.194343 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:09.194345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:09.194403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:09.194582 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:56:09.194837 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.195447 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.196359 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:09.196580 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:09.196685 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:09.196997 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:56:09.197493 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:56:09.197641 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:09.197678 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 06:56:09.197733 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:56:09.204568 6061 raft_consensus.cc:1261] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Refusing update from remote peer 6f966db3513d4d5fb735f1e598d20c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:56:09.204653 6195 raft_consensus.cc:1261] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Refusing update from remote peer 6f966db3513d4d5fb735f1e598d20c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:56:09.205353 6400 consensus_queue.cc:1035] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:56:09.205554 6402 consensus_queue.cc:1035] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:56:09.213660 6406 mvcc.cc:205] Tried to move back new op lower bound from 7298466485053730816 to 7298466484954615808. Current Snapshot: MvccSnapshot[applied={T|T < 7298466485053730816}]
I20260619 06:56:09.222834 5915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:56:09.223062 5915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:56:09.226046 5915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Sending DeleteTablet for 3 replicas of tablet 830d467569ee4e22b2898aadec309b76
I20260619 06:56:09.226953 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d467569ee4e22b2898aadec309b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:35204
I20260619 06:56:09.227010 6041 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d467569ee4e22b2898aadec309b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:50142
I20260619 06:56:09.227636 6414 tablet_replica.cc:330] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: stopping tablet replica
I20260619 06:56:09.227591 6175 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d467569ee4e22b2898aadec309b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:35680
I20260619 06:56:09.227818 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5978
I20260619 06:56:09.229218 6416 tablet_replica.cc:330] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: stopping tablet replica
I20260619 06:56:09.229606 6416 raft_consensus.cc:2229] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:56:09.229864 6414 raft_consensus.cc:2229] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:56:09.230026 6416 raft_consensus.cc:2258] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:56:09.230332 6414 raft_consensus.cc:2258] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:56:09.231822 6416 ts_tablet_manager.cc:1905] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:56:09.232219 6414 ts_tablet_manager.cc:1905] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:56:09.235180 6416 ts_tablet_manager.cc:1918] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:56:09.235318 6416 log.cc:1196] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/830d467569ee4e22b2898aadec309b76
I20260619 06:56:09.235680 6416 ts_tablet_manager.cc:1939] T 830d467569ee4e22b2898aadec309b76 P 67c03fde360b4ba4a226fd763aef14d3: Deleting consensus metadata
I20260619 06:56:09.235905 6414 ts_tablet_manager.cc:1918] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:56:09.236006 6414 log.cc:1196] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/830d467569ee4e22b2898aadec309b76
I20260619 06:56:09.236377 6414 ts_tablet_manager.cc:1939] T 830d467569ee4e22b2898aadec309b76 P 6f966db3513d4d5fb735f1e598d20c9e: Deleting consensus metadata
W20260619 06:56:09.236806 5903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:46851 (error 108)
I20260619 06:56:09.237287 5903 catalog_manager.cc:4841] TS 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): tablet 830d467569ee4e22b2898aadec309b76 (table test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]) successfully deleted
I20260619 06:56:09.237617 5901 catalog_manager.cc:4841] TS 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): tablet 830d467569ee4e22b2898aadec309b76 (table test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]) successfully deleted
I20260619 06:56:09.238301 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6112
I20260619 06:56:09.238521 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:09.238226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.0.0.1:50158)
0619 06:56:09.238358 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:09.238427 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:56:09.239837 5903 catalog_manager.cc:4584] TS cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 830d467569ee4e22b2898aadec309b76: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46851: connect: Connection refused (error 111)
I20260619 06:56:09.247493 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6246
I20260619 06:56:09.253769 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 5884
I20260619 06:56:29.261868 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--webserver_interface=127.22.113.62
--webserver_port=35607
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:29.389999 6418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:29.390360 6418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:29.390405 6418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:29.394814 6418 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:56:29.394886 6418 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:56:29.394903 6418 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:56:29.394919 6418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:56:29.394939 6418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:29.395011 6418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:29.395033 6418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:56:29.395072 6418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:56:29.400974 6418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=35607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:29.402347 6418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:29.403648 6418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:29.419932 6427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:29.419970 6425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:29.420225 6424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:29.420027 6418 server_base.cc:1034] running on GCE node
I20260619 06:56:29.421048 6418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:29.422695 6418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:29.423934 6418 hybrid_clock.cc:648] HybridClock initialized: now 1781852189423904 us; error 46 us; skew 500 ppm
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:29.427649 6418 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:56:29.428938 6418 webserver.cc:460] Webserver started at http://127.22.113.62:35607/ using document root <none> and password file <none>
I20260619 06:56:29.429742 6418 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:29.429883 6418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:29.435141 6418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:56:29.438043 6434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:29.439747 6418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:56:29.439949 6418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "4de7f71695ff4b4d95b5f194bf27bcb4"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "2c8df6460abd3d286845635618a2841e"
server_key_iv: "e1fb5eda1121c0ef66086c377345699f"
server_key_version: "encryptionkey@0"
I20260619 06:56:29.440770 6418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:29.469378 6418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:29.473237 6418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:29.473521 6418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:29.482893 6418 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40579
I20260619 06:56:29.483237 6486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40579 every 8 connection(s)
I20260619 06:56:29.484340 6418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:56:29.489107 6487 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:56:29.490043 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6418
I20260619 06:56:29.490532 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:46851
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=39995
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:56:29.491223 6487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap starting.
I20260619 06:56:29.508642 6487 log.cc:824] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:29.517293 6487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:56:29.517688 6487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap complete.
I20260619 06:56:29.521884 6487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:29.522276 6487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Initialized, Role: FOLLOWER
I20260619 06:56:29.522763 6487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:29.522879 6487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:56:29.522974 6487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:56:29.523097 6487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:56:29.525002 6487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:29.525341 6487 leader_election.cc:302] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de7f71695ff4b4d95b5f194bf27bcb4; no voters:
I20260619 06:56:29.525671 6487 leader_election.cc:288] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:56:29.525895 6491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:56:29.526316 6491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 LEADER]: Becoming Leader. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Running, Role: LEADER
I20260619 06:56:29.526662 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:29.526924 6487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:56:29.528326 6493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de7f71695ff4b4d95b5f194bf27bcb4. Latest consensus state: current_term: 2 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:29.528580 6493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:29.528370 6492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:29.528669 6492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:29.532398 6506 catalog_manager.cc:1259] Loaded cluster ID: aece56151e8a4ad6a95e92a6a0c45cd8
I20260619 06:56:29.532464 6506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: loading cluster ID for follower catalog manager: success
I20260619 06:56:29.534982 6506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: acquiring CA information for follower catalog manager: success
I20260619 06:56:29.536644 6506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:56:29.536825 6503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:56:29.539397 6503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:56:29.539945 6503 catalog_manager.cc:1259] Loaded cluster ID: aece56151e8a4ad6a95e92a6a0c45cd8
I20260619 06:56:29.540007 6503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:56:29.540684 6503 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:56:29.541184 6503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Loaded TSK: 0
I20260619 06:56:29.554716 6503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Generated new TSK 1
I20260619 06:56:29.554953 6503 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:56:29.626652 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:29.626986 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:29.627069 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:29.631845 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:29.631946 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:29.632050 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:56:29.637693 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:46851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=39995
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:29.639103 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:29.640383 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:29.651746 6514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:29.651947 6489 server_base.cc:1034] running on GCE node
W20260619 06:56:29.652070 6515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:29.651777 6517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:29.652637 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:29.653401 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:29.654608 6489 hybrid_clock.cc:648] HybridClock initialized: now 1781852189654583 us; error 49 us; skew 500 ppm
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:29.657886 6489 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:56:29.659044 6489 webserver.cc:460] Webserver started at http://127.22.113.1:39995/ using document root <none> and password file <none>
I20260619 06:56:29.659765 6489 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:29.659879 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:29.664523 6489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:56:29.667011 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:29.668542 6489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:56:29.668676 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "cc5949f652fc46df9facc33661d6d4ec"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "8d129d9f21badcff2176bd024d68435c"
server_key_iv: "812eb847736fe32fc093305d4acfac4b"
server_key_version: "encryptionkey@0"
I20260619 06:56:29.669308 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:29.683626 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:29.687402 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:29.687655 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:29.688508 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:29.689986 6531 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:56:29.694406 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:56:29.694491 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 06:56:29.694594 6489 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:56:29.696813 6489 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:56:29.696868 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 06:56:29.697211 6531 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Bootstrap starting.
I20260619 06:56:29.709504 6489 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46851
I20260619 06:56:29.709582 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46851 every 8 connection(s)
I20260619 06:56:29.710682 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:56:29.716077 6531 log.cc:824] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:29.719503 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6489
I20260619 06:56:29.720007 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:32783
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=42795
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:56:29.724378 6531 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:56:29.724812 6531 tablet_bootstrap.cc:492] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Bootstrap complete.
I20260619 06:56:29.726059 6531 ts_tablet_manager.cc:1397] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.007s
I20260619 06:56:29.730099 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.713277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:46793 (local address 127.22.113.62:40579)
0619 06:56:29.713740 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:29.713763 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:29.714143 (+ 380us) server_negotiation.cc:407] Connection header received
0619 06:56:29.715433 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:29.715444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:29.715765 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:29.716109 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:29.717543 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.718570 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.719922 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.720584 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.720785 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.723965 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:29.723970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:29.723983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:29.724014 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:29.726203 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.726817 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.726823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.726828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.726961 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.727370 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.727375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.727376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.727816 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:29.727963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:29.728745 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 06:56:29.729033 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":158,"threads_started":1}
I20260619 06:56:29.730099 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.713257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:46793)
0619 06:56:29.713938 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:29.713984 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:29.714939 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:29.716170 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:29.716178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:29.716730 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:29.717347 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.717365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.718745 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.718748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.719797 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.719807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.720752 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.720756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.720943 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.721697 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:29.721722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:29.723749 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:29.726325 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.726332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.726353 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.726690 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.727069 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.727072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.727074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.727223 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.728015 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:29.728020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:29.728313 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 06:56:29.728619 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 06:56:29.729027 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":193,"threads_started":1}
I20260619 06:56:29.730198 6531 raft_consensus.cc:348] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:29.730782 6531 raft_consensus.cc:729] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5949f652fc46df9facc33661d6d4ec, State: Initialized, Role: FOLLOWER
I20260619 06:56:29.731343 6531 consensus_queue.cc:260] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } }
I20260619 06:56:29.732540 6531 ts_tablet_manager.cc:1428] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:56:29.733403 6639 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:29.733703 6639 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:29.734274 6639 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:29.737088 6451 ts_manager.cc:194] Registered new tserver with Master: cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851)
I20260619 06:56:29.741241 6451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:46793
I20260619 06:56:29.744477 6639 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.1@KRBTEST.COM
I20260619 06:56:29.754575 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.740514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.0.0.1:37828)
0619 06:56:29.741440 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:29.741517 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:29.741744 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:29.742583 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:29.742589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:29.743107 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:29.743304 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.743315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.744951 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.744955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.745770 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.745782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.746708 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.746712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.746826 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.747542 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:29.747554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:29.749391 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:29.751887 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.751901 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.751910 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.752320 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.752829 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.752854 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.752857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.752991 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.753732 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:29.753740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:29.753898 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 06:56:29.754068 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 06:56:29.754342 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":133,"threads_started":1}
I20260619 06:56:29.754575 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.740594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.22.113.1:46851)
0619 06:56:29.741044 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:29.741056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:29.741659 (+ 603us) server_negotiation.cc:407] Connection header received
0619 06:56:29.741953 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:29.741959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:29.742203 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:29.742360 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:29.743459 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.744775 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.745915 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.746428 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.746531 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.749655 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:29.749666 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:29.749681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:29.749718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:29.751733 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.752519 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.752529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.752533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.752655 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.753126 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.753133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.753137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.753581 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:29.753770 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:29.754170 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 06:56:29.754374 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":130,"threads_started":1}
I20260619 06:56:29.756428 6573 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d467569ee4e22b2898aadec309b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:09 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37828
I20260619 06:56:29.757966 6648 tablet_replica.cc:330] stopping tablet replica
I20260619 06:56:29.758464 6648 raft_consensus.cc:2229] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:56:29.758841 6648 raft_consensus.cc:2258] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:56:29.760344 6648 ts_tablet_manager.cc:1905] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:56:29.764988 6648 ts_tablet_manager.cc:1918] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:56:29.765144 6648 log.cc:1196] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal/wals/830d467569ee4e22b2898aadec309b76
I20260619 06:56:29.765422 6648 ts_tablet_manager.cc:1939] T 830d467569ee4e22b2898aadec309b76 P cc5949f652fc46df9facc33661d6d4ec: Deleting consensus metadata
I20260619 06:56:29.767696 6438 catalog_manager.cc:4841] TS cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851): tablet 830d467569ee4e22b2898aadec309b76 (table test-table [id=c8457ff8ab0f4ae291e7f980e13bb281]) successfully deleted
W20260619 06:56:29.848783 6644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:29.849040 6644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:29.849080 6644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:29.853189 6644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:29.853255 6644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:29.853348 6644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:56:29.858590 6644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:32783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=42795
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:29.859694 6644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:29.860723 6644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:29.871161 6656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:29.871416 6644 server_base.cc:1034] running on GCE node
W20260619 06:56:29.871181 6653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:29.871160 6654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:29.872114 6644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:29.872781 6644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:29.873989 6644 hybrid_clock.cc:648] HybridClock initialized: now 1781852189873967 us; error 41 us; skew 500 ppm
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:29.877142 6644 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:56:29.878096 6644 webserver.cc:460] Webserver started at http://127.22.113.2:42795/ using document root <none> and password file <none>
I20260619 06:56:29.878646 6644 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:29.878717 6644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:29.882783 6644 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:56:29.885138 6663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:29.886590 6644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:56:29.886747 6644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "67c03fde360b4ba4a226fd763aef14d3"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "3639b2587d34829f3c9376701bec1898"
server_key_iv: "c5e08146540d43f7ccff0b553a0f0466"
server_key_version: "encryptionkey@0"
I20260619 06:56:29.887374 6644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:29.906893 6644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:29.910630 6644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:29.910880 6644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:29.911679 6644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:29.912640 6644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:29.912691 6644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:29.912734 6644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:29.912798 6644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:29.924181 6644 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:32783
I20260619 06:56:29.924275 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:32783 every 8 connection(s)
I20260619 06:56:29.925426 6644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:56:29.931910 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6644
I20260619 06:56:29.932389 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:38605
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=36865
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:56:29 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852189, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:56:29.942552 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.927756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:39519 (local address 127.22.113.62:40579)
0619 06:56:29.927984 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:29.927989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:29.928982 (+ 993us) server_negotiation.cc:407] Connection header received
0619 06:56:29.930611 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:29.930615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:29.930688 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:29.930796 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:29.932321 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.932985 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.934711 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.935009 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.935179 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.938088 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:29.938096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:29.938102 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:29.938140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:29.940071 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.940642 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.940647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.940650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.940745 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:29.941118 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:29.941123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:29.941127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:29.941416 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:29.941546 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:29.942128 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 06:56:29.942296 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:56:29.943529 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:29.928200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:39519)
0619 06:56:29.928768 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:29.928817 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:29.930354 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:29.931018 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:29.931025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:29.931501 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:29.932166 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.932183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.933116 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.933119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.934523 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:29.934537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:29.935221 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:29.935224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:29.935469 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:29.936203 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:29.936220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:29.937869 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:29.940209 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.940214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.940227 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.940526 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.940863 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:29.940865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:29.940867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:29.941013 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:29.941573 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:29.941579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:29.941862 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 06:56:29.942036 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:56:29.942335 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":164,"threads_started":1}
I20260619 06:56:29.944993 6777 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:29.945330 6777 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:29.946022 6777 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:29.947314 6451 ts_manager.cc:194] Registered new tserver with Master: 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
I20260619 06:56:29.948067 6451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:39519
W20260619 06:56:30.047226 6780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:30.047551 6780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:30.047595 6780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:30.051680 6780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:30.051741 6780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:30.051836 6780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:56:30.056944 6780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:38605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=36865
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:30.057893 6780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:30.058909 6780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:30.068848 6785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:30.068864 6788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:30.069171 6780 server_base.cc:1034] running on GCE node
W20260619 06:56:30.068848 6786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:30.069630 6780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:30.070282 6780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:30.071480 6780 hybrid_clock.cc:648] HybridClock initialized: now 1781852190071458 us; error 43 us; skew 500 ppm
Jun 19 06:56:30 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852190, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:30.074369 6780 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:56:30.075304 6780 webserver.cc:460] Webserver started at http://127.22.113.3:36865/ using document root <none> and password file <none>
I20260619 06:56:30.075873 6780 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:30.075973 6780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:30.079816 6780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:56:30.081916 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:30.083209 6780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:56:30.083448 6780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6f966db3513d4d5fb735f1e598d20c9e"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "a6769eb2b93e3498dfba2b2377a828b6"
server_key_iv: "6c766a93719a156d095d353ff3bbba0e"
server_key_version: "encryptionkey@0"
I20260619 06:56:30.083964 6780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:30.107383 6780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:30.110466 6780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:30.110678 6780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:30.111325 6780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:30.112196 6780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:30.112278 6780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:30.112350 6780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:30.112387 6780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:30.122604 6780 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38605
I20260619 06:56:30.122641 6908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38605 every 8 connection(s)
I20260619 06:56:30.123664 6780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:56:30.130126 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6780
Jun 19 06:56:30 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852190, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:56:30.140651 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.131053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.22.113.62:40579)
0619 06:56:30.131674 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.131678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.131691 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:56:30.131724 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.131726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.131766 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.131860 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:30.132812 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.133542 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.134845 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.135207 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.135349 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.136735 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:30.136740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:30.136744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:30.136793 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:30.138563 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.139096 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.139099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.139100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.139303 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.139624 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.139626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.139628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.139897 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:30.140135 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.140315 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.140443 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":544,"spinlock_wait_cycles":4864,"thread_start_us":104,"threads_started":1}
I20260619 06:56:30.140959 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.125948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:36503 (local address 127.22.113.62:40579)
0619 06:56:30.126115 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.126120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.126980 (+ 860us) server_negotiation.cc:407] Connection header received
0619 06:56:30.128477 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.128481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.128553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.128663 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:30.129932 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.130784 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.132578 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.132863 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.132975 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.136275 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:30.136278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:30.136280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:30.136303 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:30.138294 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.139090 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.139094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.139096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.139282 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.139729 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.139732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.139733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.139897 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:30.140010 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.140695 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.140837 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:56:30.141906 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.126275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:36503)
0619 06:56:30.126812 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.126863 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:30.128222 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:30.128876 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:30.128883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:30.129232 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:30.129704 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.129717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.130957 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.130963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.132433 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.132445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.133025 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.133028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.133172 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.133901 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:30.133921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:30.136077 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:30.138448 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:30.138454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:30.138470 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:30.138847 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:30.139407 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:30.139411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:30.139413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:30.139613 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:30.140033 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:30.140040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:30.140391 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 06:56:30.140613 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 06:56:30.140934 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260619 06:56:30.143239 6909 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40579
I20260619 06:56:30.143615 6909 heartbeater.cc:464] Registering TS with master...
I20260619 06:56:30.144200 6909 heartbeater.cc:511] Master 127.22.113.62:40579 requested a full tablet report, sending...
I20260619 06:56:30.145368 6451 ts_manager.cc:194] Registered new tserver with Master: 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605)
I20260619 06:56:30.145970 6451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:36503
I20260619 06:56:30.155645 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:56:30.160218 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.156416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.22.113.62:40579)
0619 06:56:30.156559 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.156562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.156780 (+ 218us) server_negotiation.cc:407] Connection header received
0619 06:56:30.156885 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.156888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.157003 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.157119 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:56:30.157358 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.157933 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.158892 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.159114 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.159212 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.159669 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:56:30.159846 (+ 177us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 06:56:30.160087 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40846: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 06:56:30.160312 6641 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40846: token expired
W20260619 06:56:30.160794 6380 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.113.62:40579)
I20260619 06:56:30.168342 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.161251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.22.113.62:40579)
0619 06:56:30.161392 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.161398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.161416 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:56:30.161582 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.161585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.161633 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.161717 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:30.162357 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.162856 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.163797 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.164012 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.164117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.165487 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:30.165491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:30.165495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:30.165529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:30.166740 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.167135 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.167139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.167141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.167222 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:30.167611 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:30.167620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:30.167622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:30.167788 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:30.167898 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.168130 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.168230 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:56:30.171212 6451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:56:30.175447 6451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:56:30.184424 6573 tablet_service.cc:1467] Processing CreateTablet for tablet 150b331df7c1452f96de272268ddb7ee (DEFAULT_TABLE table=test-table [id=2e93b7ece2a64730b04cfdfff4076b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:30.185117 6573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b331df7c1452f96de272268ddb7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:30.188089 6531 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Bootstrap starting.
I20260619 06:56:30.188472 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.183001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38605 (local address 127.0.0.1:38316)
0619 06:56:30.183167 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.183182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:30.183297 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:30.184320 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:30.184323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:30.184341 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:30.184446 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.184456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.185831 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.185837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.186782 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.186792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.187922 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.187926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.188039 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.188108 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:56:30.188170 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 06:56:30.188287 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:56:30.188993 6531 tablet_bootstrap.cc:654] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:30.189319 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.183224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.22.113.3:38605)
0619 06:56:30.183670 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.183689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.183721 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:56:30.183825 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.183831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.184128 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.184332 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:30.184563 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.185714 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.186909 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.187789 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.188806 (+ 1017us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.188974 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.189022 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.189157 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260619 06:56:30.189827 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.183235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:32783 (local address 127.0.0.1:59366)
0619 06:56:30.183583 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.183600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:30.183701 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:30.184891 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:30.184894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:30.184910 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:30.184991 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.185000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.186538 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.186541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.187949 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.187958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.189170 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.189174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.189287 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.189316 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:56:30.189602 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 06:56:30.189692 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
I20260619 06:56:30.190336 6531 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: No bootstrap required, opened a new log
I20260619 06:56:30.190368 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.183803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.22.113.2:32783)
0619 06:56:30.184279 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.184291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.184314 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:56:30.184393 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.184398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.184581 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.184734 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:30.185160 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.186397 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.188085 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.188992 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.189108 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.189302 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.189919 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.190032 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":199,"threads_started":1}
I20260619 06:56:30.190435 6531 ts_tablet_manager.cc:1397] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:56:30.190876 6531 raft_consensus.cc:348] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.190999 6531 raft_consensus.cc:374] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:30.191035 6531 raft_consensus.cc:729] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5949f652fc46df9facc33661d6d4ec, State: Initialized, Role: FOLLOWER
I20260619 06:56:30.191164 6531 consensus_queue.cc:260] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.191443 6531 ts_tablet_manager.cc:1428] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:56:30.193408 6711 tablet_service.cc:1467] Processing CreateTablet for tablet 150b331df7c1452f96de272268ddb7ee (DEFAULT_TABLE table=test-table [id=2e93b7ece2a64730b04cfdfff4076b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:30.193399 6843 tablet_service.cc:1467] Processing CreateTablet for tablet 150b331df7c1452f96de272268ddb7ee (DEFAULT_TABLE table=test-table [id=2e93b7ece2a64730b04cfdfff4076b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:56:30.194417 6711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b331df7c1452f96de272268ddb7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:30.194567 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150b331df7c1452f96de272268ddb7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:56:30.201153 6922 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Bootstrap starting.
I20260619 06:56:30.203678 6922 tablet_bootstrap.cc:654] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:30.203729 6923 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Bootstrap starting.
I20260619 06:56:30.204581 6922 log.cc:824] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:30.205713 6923 tablet_bootstrap.cc:654] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Neither blocks nor log segments found. Creating new log.
I20260619 06:56:30.206665 6923 log.cc:824] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:30.206763 6922 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: No bootstrap required, opened a new log
I20260619 06:56:30.207000 6922 ts_tablet_manager.cc:1397] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 06:56:30.208452 6923 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: No bootstrap required, opened a new log
I20260619 06:56:30.208648 6923 ts_tablet_manager.cc:1397] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:56:30.210564 6922 raft_consensus.cc:348] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.210899 6922 raft_consensus.cc:374] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:30.210963 6922 raft_consensus.cc:729] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c03fde360b4ba4a226fd763aef14d3, State: Initialized, Role: FOLLOWER
I20260619 06:56:30.211498 6922 consensus_queue.cc:260] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
W20260619 06:56:30.212181 6640 tablet.cc:2369] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:56:30.212224 6777 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
I20260619 06:56:30.212649 6922 ts_tablet_manager.cc:1428] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 06:56:30.213246 6923 raft_consensus.cc:348] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.213563 6923 raft_consensus.cc:374] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:56:30.213624 6923 raft_consensus.cc:729] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f966db3513d4d5fb735f1e598d20c9e, State: Initialized, Role: FOLLOWER
I20260619 06:56:30.214049 6923 consensus_queue.cc:260] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.214783 6909 heartbeater.cc:503] Master 127.22.113.62:40579 was elected leader, sending a full tablet report...
I20260619 06:56:30.215577 6923 ts_tablet_manager.cc:1428] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:56:30.282500 6927 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:30.282841 6927 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.284332 6927 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783), cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851)
I20260619 06:56:30.289019 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.284796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46915 (local address 127.22.113.2:32783)
0619 06:56:30.284959 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.284966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.285222 (+ 256us) server_negotiation.cc:407] Connection header received
0619 06:56:30.285294 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.285297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.285360 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.285430 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:30.285840 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.286602 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.287934 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.288379 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.288560 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.288615 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.288853 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.288913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:56:30.289088 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.284738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:32783 (local address 127.22.113.3:46915)
0619 06:56:30.284913 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.284932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:30.285110 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:30.285573 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:30.285576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:30.285592 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:30.285725 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.285736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.286753 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.286755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.287797 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.287806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.288588 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.288592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.288715 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.288746 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:56:30.288831 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:56:30.288916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":4}
I20260619 06:56:30.289644 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.285039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.22.113.3:44435)
0619 06:56:30.285499 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.285521 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:30.285713 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:30.286189 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:30.286192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:30.286209 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:56:30.286332 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.286337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.287171 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.287176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.288193 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:30.288203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.289161 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.289165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:30.289286 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.289316 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:56:30.289408 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:56:30.289506 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":135,"threads_started":1}
I20260619 06:56:30.289647 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.285159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:44435 (local address 127.22.113.1:46851)
0619 06:56:30.285444 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.285448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.285611 (+ 163us) server_negotiation.cc:407] Connection header received
0619 06:56:30.285928 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.285931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.286025 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.286161 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:56:30.286453 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.287059 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.288319 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.289001 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.289136 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.289330 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.289427 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.289515 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260619 06:56:30.290256 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b331df7c1452f96de272268ddb7ee" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c03fde360b4ba4a226fd763aef14d3" is_pre_election: true
I20260619 06:56:30.290432 6593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b331df7c1452f96de272268ddb7ee" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5949f652fc46df9facc33661d6d4ec" is_pre_election: true
I20260619 06:56:30.290666 6731 raft_consensus.cc:2454] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 0.
I20260619 06:56:30.290685 6593 raft_consensus.cc:2454] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 0.
I20260619 06:56:30.291285 6799 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e; no voters:
I20260619 06:56:30.291579 6927 raft_consensus.cc:2781] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:56:30.291669 6927 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:56:30.291715 6927 raft_consensus.cc:3037] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:30.292992 6927 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.293388 6927 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 election: Requested vote from peers 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783), cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851)
I20260619 06:56:30.293844 6593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b331df7c1452f96de272268ddb7ee" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5949f652fc46df9facc33661d6d4ec"
I20260619 06:56:30.293905 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150b331df7c1452f96de272268ddb7ee" candidate_uuid: "6f966db3513d4d5fb735f1e598d20c9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c03fde360b4ba4a226fd763aef14d3"
I20260619 06:56:30.293983 6593 raft_consensus.cc:3037] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:30.294153 6731 raft_consensus.cc:3037] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:56:30.295194 6593 raft_consensus.cc:2454] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 1.
I20260619 06:56:30.295496 6731 raft_consensus.cc:2454] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f966db3513d4d5fb735f1e598d20c9e in term 1.
I20260619 06:56:30.295547 6799 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f966db3513d4d5fb735f1e598d20c9e, cc5949f652fc46df9facc33661d6d4ec; no voters:
I20260619 06:56:30.295786 6927 raft_consensus.cc:2781] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:56:30.296095 6927 raft_consensus.cc:686] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 1 LEADER]: Becoming Leader. State: Replica: 6f966db3513d4d5fb735f1e598d20c9e, State: Running, Role: LEADER
I20260619 06:56:30.296510 6927 consensus_queue.cc:237] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:30.300005 6451 catalog_manager.cc:5495] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3). New cstate: current_term: 1 leader_uuid: "6f966db3513d4d5fb735f1e598d20c9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } health_report { overall_health: UNKNOWN } } }
I20260619 06:56:30.318593 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:30.314605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.22.113.3:38605)
0619 06:56:30.314739 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:30.314743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:30.314853 (+ 110us) server_negotiation.cc:407] Connection header received
0619 06:56:30.315015 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:30.315017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:30.315075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:30.315174 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:56:30.315536 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.316060 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.317035 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:30.317285 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:30.317389 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:30.317731 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:56:30.318185 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:56:30.318293 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:30.318430 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:56:30.318478 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:56:30.325585 6731 raft_consensus.cc:1261] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Refusing update from remote peer 6f966db3513d4d5fb735f1e598d20c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:56:30.325608 6593 raft_consensus.cc:1261] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Refusing update from remote peer 6f966db3513d4d5fb735f1e598d20c9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:56:30.326356 6927 consensus_queue.cc:1035] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:56:30.326665 6930 consensus_queue.cc:1035] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:56:30.338882 6935 mvcc.cc:205] Tried to move back new op lower bound from 7298466571564822528 to 7298466571457695744. Current Snapshot: MvccSnapshot[applied={T|T < 7298466571564822528}]
I20260619 06:56:30.345623 6450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:56:30.345837 6450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:56:30.348582 6450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Sending DeleteTablet for 3 replicas of tablet 150b331df7c1452f96de272268ddb7ee
I20260619 06:56:30.349344 6843 tablet_service.cc:1514] Processing DeleteTablet for tablet 150b331df7c1452f96de272268ddb7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:30 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:38316
I20260619 06:56:30.349467 6573 tablet_service.cc:1514] Processing DeleteTablet for tablet 150b331df7c1452f96de272268ddb7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:30 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:37828
I20260619 06:56:30.349455 6711 tablet_service.cc:1514] Processing DeleteTablet for tablet 150b331df7c1452f96de272268ddb7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:56:30 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59366
I20260619 06:56:30.349918 6943 tablet_replica.cc:330] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: stopping tablet replica
I20260619 06:56:30.349921 6942 tablet_replica.cc:330] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: stopping tablet replica
I20260619 06:56:30.350257 6943 raft_consensus.cc:2229] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:56:30.350245 6942 raft_consensus.cc:2229] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:56:30.350358 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6489
I20260619 06:56:30.350576 6942 raft_consensus.cc:2258] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:56:30.352420 6942 ts_tablet_manager.cc:1905] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:56:30.355144 6942 ts_tablet_manager.cc:1918] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:56:30.355283 6942 log.cc:1196] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/150b331df7c1452f96de272268ddb7ee
I20260619 06:56:30.355645 6942 ts_tablet_manager.cc:1939] T 150b331df7c1452f96de272268ddb7ee P 67c03fde360b4ba4a226fd763aef14d3: Deleting consensus metadata
I20260619 06:56:30.355773 6943 raft_consensus.cc:2258] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:56:30.356962 6438 catalog_manager.cc:4841] TS 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): tablet 150b331df7c1452f96de272268ddb7ee (table test-table [id=2e93b7ece2a64730b04cfdfff4076b3f]) successfully deleted
I20260619 06:56:30.357746 6943 ts_tablet_manager.cc:1905] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:56:30.357877 6438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:46851 (error 108)
W20260619 06:56:30.359308 6919 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:56:30.358947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46851 (local address 127.0.0.1:37844)
0619 06:56:30.359093 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:30.359171 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46851: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 06:56:30.359467 6438 catalog_manager.cc:4584] TS cc5949f652fc46df9facc33661d6d4ec (127.22.113.1:46851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 150b331df7c1452f96de272268ddb7ee: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46851: connect: Connection reset by peer (error 104)
I20260619 06:56:30.360486 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6644
I20260619 06:56:30.361789 6943 ts_tablet_manager.cc:1918] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:56:30.361940 6943 log.cc:1196] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/150b331df7c1452f96de272268ddb7ee
I20260619 06:56:30.362305 6943 ts_tablet_manager.cc:1939] T 150b331df7c1452f96de272268ddb7ee P 6f966db3513d4d5fb735f1e598d20c9e: Deleting consensus metadata
I20260619 06:56:30.363651 6436 catalog_manager.cc:4841] TS 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): tablet 150b331df7c1452f96de272268ddb7ee (table test-table [id=2e93b7ece2a64730b04cfdfff4076b3f]) successfully deleted
I20260619 06:56:30.367213 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6780
I20260619 06:56:30.374012 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 6418
I20260619 06:56:50.381354 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--webserver_interface=127.22.113.62
--webserver_port=35607
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:50.493027 6945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:50.493362 6945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:50.493446 6945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:50.497481 6945 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:56:50.497553 6945 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:56:50.497589 6945 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:56:50.497622 6945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:56:50.497665 6945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:50.497720 6945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:50.497757 6945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:56:50.497793 6945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:56:50.503322 6945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=35607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:50.504453 6945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:50.505472 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:50.514521 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.514540 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.514540 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:50.514956 6945 server_base.cc:1034] running on GCE node
I20260619 06:56:50.515604 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:50.516777 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:50.517967 6945 hybrid_clock.cc:648] HybridClock initialized: now 1781852210517952 us; error 36 us; skew 500 ppm
Jun 19 06:56:50 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:50.520926 6945 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:56:50.521872 6945 webserver.cc:460] Webserver started at http://127.22.113.62:35607/ using document root <none> and password file <none>
I20260619 06:56:50.522449 6945 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:50.522538 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:50.526836 6945 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:56:50.529553 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:50.530958 6945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:56:50.531105 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "4de7f71695ff4b4d95b5f194bf27bcb4"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "2c8df6460abd3d286845635618a2841e"
server_key_iv: "e1fb5eda1121c0ef66086c377345699f"
server_key_version: "encryptionkey@0"
I20260619 06:56:50.531741 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:50.548372 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:50.551472 6945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:50.551697 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:50.559404 6945 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40579
I20260619 06:56:50.559696 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40579 every 8 connection(s)
I20260619 06:56:50.560576 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:56:50.565830 7013 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:56:50.567890 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap starting.
I20260619 06:56:50.569291 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 6945
I20260619 06:56:50.569905 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:46851
--local_ip_for_outbound_sockets=127.22.113.1
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=39995
--webserver_interface=127.22.113.1
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:56:50.589171 7013 log.cc:824] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:50.601598 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:56:50.602067 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Bootstrap complete.
I20260619 06:56:50.606777 7013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:50.607210 7013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Initialized, Role: FOLLOWER
I20260619 06:56:50.607838 7013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:50.607965 7013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:56:50.608058 7013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:56:50.608176 7013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:56:50.610376 7013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:50.610745 7013 leader_election.cc:302] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de7f71695ff4b4d95b5f194bf27bcb4; no voters:
I20260619 06:56:50.611114 7013 leader_election.cc:288] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:56:50.611207 7017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:56:50.611526 7017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [term 3 LEADER]: Becoming Leader. State: Replica: 4de7f71695ff4b4d95b5f194bf27bcb4, State: Running, Role: LEADER
I20260619 06:56:50.611899 7017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } }
I20260619 06:56:50.612550 7013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:56:50.613583 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:50.613832 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:50.614367 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de7f71695ff4b4d95b5f194bf27bcb4. Latest consensus state: current_term: 3 leader_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de7f71695ff4b4d95b5f194bf27bcb4" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40579 } } }
I20260619 06:56:50.614516 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:56:50.614621 7026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:56:50.620458 7026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:56:50.621182 7026 catalog_manager.cc:1259] Loaded cluster ID: aece56151e8a4ad6a95e92a6a0c45cd8
I20260619 06:56:50.621250 7026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:56:50.623993 7026 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:56:50.624765 7026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Loaded TSK: 0
I20260619 06:56:50.624867 7026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4de7f71695ff4b4d95b5f194bf27bcb4: Loaded TSK: 1
F20260619 06:56:50.625849 7026 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38fedaf9ad google::LogMessage::Fail() at ??:0
@ 0x7f38fedb288a google::LogMessage::SendToLog() at ??:0
@ 0x7f38fedaf4de google::LogMessage::Flush() at ??:0
@ 0x7f38fedb0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3901a7318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3901a6bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f39097147bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f39096ee99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f39096ef178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3909730df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f390974f82b std::function<>::operator()() at ??:0
@ 0x7f39096eeb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f39096eff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39096ec579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3909730640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f39010013d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3901001c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f39010033f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3900ff2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39026366db start_thread at ??:0
@ 0x7f38fd8ad61f clone at ??:0
W20260619 06:56:50.718899 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:50.719156 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:50.719231 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:50.723471 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:50.723528 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:50.723625 7015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:56:50.729256 7015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:46851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=39995
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:50.730352 7015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:50.731413 7015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:50.741395 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.741442 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.741375 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:50.742229 7015 server_base.cc:1034] running on GCE node
I20260619 06:56:50.742666 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:50.743371 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:50.744549 7015 hybrid_clock.cc:648] HybridClock initialized: now 1781852210744529 us; error 44 us; skew 500 ppm
Jun 19 06:56:50 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:50.747615 7015 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:56:50.748553 7015 webserver.cc:460] Webserver started at http://127.22.113.1:39995/ using document root <none> and password file <none>
I20260619 06:56:50.749135 7015 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:50.749221 7015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:50.753167 7015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:56:50.755594 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:50.757179 7015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:56:50.757344 7015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "cc5949f652fc46df9facc33661d6d4ec"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "8d129d9f21badcff2176bd024d68435c"
server_key_iv: "812eb847736fe32fc093305d4acfac4b"
server_key_version: "encryptionkey@0"
I20260619 06:56:50.757894 7015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:50.778673 7015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:50.781857 7015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:50.782050 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:50.782611 7015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:50.783860 7077 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:56:50.787760 7015 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:56:50.787813 7015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260619 06:56:50.787879 7015 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:56:50.789404 7015 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:56:50.789462 7015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 06:56:50.789683 7077 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Bootstrap starting.
I20260619 06:56:50.801261 7015 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46851
I20260619 06:56:50.801666 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46851 every 8 connection(s)
I20260619 06:56:50.802593 7015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:56:50.808549 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7015
I20260619 06:56:50.809197 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:32783
--local_ip_for_outbound_sockets=127.22.113.2
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=42795
--webserver_interface=127.22.113.2
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:56:50.810407 7077 log.cc:824] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:56:50.817170 7077 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:56:50.817588 7077 tablet_bootstrap.cc:492] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Bootstrap complete.
I20260619 06:56:50.818908 7077 ts_tablet_manager.cc:1397] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20260619 06:56:50.822940 7077 raft_consensus.cc:348] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:50.823627 7077 raft_consensus.cc:729] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5949f652fc46df9facc33661d6d4ec, State: Initialized, Role: FOLLOWER
I20260619 06:56:50.824198 7077 consensus_queue.cc:260] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:50.825539 7077 ts_tablet_manager.cc:1428] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260619 06:56:50.857409 7187 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:56:50.805789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:46913)
0619 06:56:50.806526 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:50.806599 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:50.807837 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:50.856228 (+ 48391us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":201,"threads_started":1}
W20260619 06:56:50.858050 7185 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:40579: Client connection negotiation failed: client connection to 127.22.113.62:40579: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:56:50.858987 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:50.858688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:41191)
0619 06:56:50.858856 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:50.858927 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:56:50.859999 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:50.859592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:34361)
0619 06:56:50.859762 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:50.859839 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:56:50.860340 7185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:56:50.926163 7189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:50.926438 7189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:50.926481 7189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:50.930496 7189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:50.930553 7189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:50.930610 7189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:56:50.935653 7189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:32783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=42795
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:50.936594 7189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:50.937520 7189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:50.947208 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.947220 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:50.947220 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:50.947592 7189 server_base.cc:1034] running on GCE node
I20260619 06:56:50.948238 7189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:50.949067 7189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:50.950335 7189 hybrid_clock.cc:648] HybridClock initialized: now 1781852210950268 us; error 39 us; skew 500 ppm
Jun 19 06:56:50 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:50.953694 7189 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:56:50.954717 7189 webserver.cc:460] Webserver started at http://127.22.113.2:42795/ using document root <none> and password file <none>
I20260619 06:56:50.955442 7189 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:50.955535 7189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:50.960155 7189 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:56:50.962940 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:50.964594 7189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:56:50.964761 7189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "67c03fde360b4ba4a226fd763aef14d3"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "3639b2587d34829f3c9376701bec1898"
server_key_iv: "c5e08146540d43f7ccff0b553a0f0466"
server_key_version: "encryptionkey@0"
I20260619 06:56:50.965456 7189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:50.987828 7189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:50.991679 7189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:50.991914 7189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:50.992681 7189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:50.993644 7189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:50.993697 7189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:50.993742 7189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:50.993757 7189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:51.004664 7189 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:32783
I20260619 06:56:51.004760 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:32783 every 8 connection(s)
I20260619 06:56:51.005896 7189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:56:51.010670 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.008716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:51847)
0619 06:56:51.009280 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.009416 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
W20260619 06:56:51.011296 7319 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:40579: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
I20260619 06:56:51.011986 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.011754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:48535)
0619 06:56:51.011876 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.011932 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:56:51.012627 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7189
I20260619 06:56:51.012775 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.012554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:50571)
0619 06:56:51.012652 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.012694 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:56:51.013027 7319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:56:51.013252 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:38605
--local_ip_for_outbound_sockets=127.22.113.3
--tserver_master_addrs=127.22.113.62:40579
--webserver_port=36865
--webserver_interface=127.22.113.3
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:56:51.143422 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:56:51.143841 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:56:51.143930 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:56:51.148291 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:56:51.148356 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:56:51.148451 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:56:51.154253 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:38605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=36865
--tserver_master_addrs=127.22.113.62:40579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:56:51.155624 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:56:51.156839 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:56:51.168202 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:56:51.168239 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:51.168540 7322 server_base.cc:1034] running on GCE node
W20260619 06:56:51.168684 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:56:51.169203 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:56:51.170012 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:56:51.171190 7322 hybrid_clock.cc:648] HybridClock initialized: now 1781852211171120 us; error 87 us; skew 500 ppm
Jun 19 06:56:51 dist-test-slave-g8zg krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852211, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:56:51.174095 7322 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:56:51.174985 7322 webserver.cc:460] Webserver started at http://127.22.113.3:36865/ using document root <none> and password file <none>
I20260619 06:56:51.175668 7322 fs_manager.cc:359] Metadata directory not provided
I20260619 06:56:51.175735 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:56:51.179920 7322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:56:51.182243 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:51.183960 7322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:56:51.184103 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "6f966db3513d4d5fb735f1e598d20c9e"
format_stamp: "Formatted at 2026-06-19 06:56:08 on dist-test-slave-g8zg"
server_key: "a6769eb2b93e3498dfba2b2377a828b6"
server_key_iv: "6c766a93719a156d095d353ff3bbba0e"
server_key_version: "encryptionkey@0"
I20260619 06:56:51.184728 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:56:51.196667 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:56:51.199687 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:56:51.199889 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:56:51.200513 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:56:51.201377 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:56:51.201445 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:51.201511 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:56:51.201556 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:56:51.211743 7322 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:38605
I20260619 06:56:51.211812 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:38605 every 8 connection(s)
I20260619 06:56:51.212858 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:56:51.217542 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.215497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:47015)
0619 06:56:51.216025 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.216175 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
W20260619 06:56:51.218161 7451 heartbeater.cc:644] Failed to heartbeat to 127.22.113.62:40579 (0 consecutive failures): Network error: Failed to ping master at 127.22.113.62:40579: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
I20260619 06:56:51.218900 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.218651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:59333)
0619 06:56:51.218768 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.218845 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:56:51.219653 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.219445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:35979)
0619 06:56:51.219546 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.219595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:56:51.219842 7451 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:56:51.221136 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7322
I20260619 06:56:51.861581 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:51.861037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:50889)
0619 06:56:51.861413 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:51.861476 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260619 06:56:52.014245 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.013720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:58481)
0619 06:56:52.014029 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:52.014108 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260619 06:56:52.220994 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.220435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:36837)
0619 06:56:52.220771 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:52.220837 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260619 06:56:52.531215 7458 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:52.531533 7458 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:52.532846 7458 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
Jun 19 06:56:52 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.3@KRBTEST.COM
Jun 19 06:56:52 dist-test-slave-g8zg krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.2@KRBTEST.COM
I20260619 06:56:52.548564 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.533545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:36403 (local address 127.22.113.2:32783)
0619 06:56:52.533948 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:52.533971 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:52.534032 (+ 61us) server_negotiation.cc:407] Connection header received
0619 06:56:52.534890 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:52.534897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:52.535413 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:52.535769 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:52.537186 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.538310 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.539521 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.540035 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.540223 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:52.543758 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:52.543763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:52.543776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:52.543804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:52.545831 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546332 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:52.546340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:52.546345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:52.546508 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546797 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:52.546801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:52.546803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:52.547303 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:52.547450 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:52.548022 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 06:56:52.548334 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260619 06:56:52.548610 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.533187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:38605 (local address 127.22.113.1:43065)
0619 06:56:52.533949 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:52.533964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:52.534136 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:52.535735 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:52.535742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:52.536456 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:52.536986 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:52.537003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.538343 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.538347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:52.539393 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:52.539402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.540129 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.540133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:52.540341 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:52.541241 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:52.541254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:52.543305 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:52.545656 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:52.545662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:52.545677 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:52.545971 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:52.546459 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:52.546465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:52.546623 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:52.547426 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:52.547433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:52.547717 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 06:56:52.547916 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 06:56:52.548309 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":449,"spinlock_wait_cycles":25728,"thread_start_us":78,"threads_started":1}
I20260619 06:56:52.548669 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.533187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:32783 (local address 127.22.113.1:36403)
0619 06:56:52.533687 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:52.533708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:56:52.533853 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:56:52.535736 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:56:52.535744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:56:52.536470 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:56:52.536987 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:52.537003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.538458 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.538465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:52.539395 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:56:52.539406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.540173 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.540177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:56:52.540359 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:52.541172 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:56:52.541207 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:56:52.543559 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:56:52.545964 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:52.545968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:52.545970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:52.546190 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:52.546610 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:56:52.546617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:56:52.546704 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:56:52.547424 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:56:52.547432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:56:52.547717 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 06:56:52.547915 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 06:56:52.548409 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":198,"spinlock_wait_cycles":8704,"thread_start_us":95,"threads_started":1}
I20260619 06:56:52.549441 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.533344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:43065 (local address 127.22.113.3:38605)
0619 06:56:52.533829 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 06:56:52.533857 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:56:52.534081 (+ 224us) server_negotiation.cc:407] Connection header received
0619 06:56:52.534904 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:56:52.534909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:56:52.535356 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:56:52.535664 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:56:52.537168 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.538203 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.539508 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:56:52.539997 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:56:52.540184 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:56:52.543528 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:56:52.543534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:56:52.543545 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:56:52.543573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:56:52.545528 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546126 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:52.546132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:52.546138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:52.546299 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:56:52.546734 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:56:52.546739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:56:52.546742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:56:52.547236 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:56:52.547454 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:56:52.548063 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 06:56:52.548376 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":137,"threads_started":1}
W20260619 06:56:52.551676 7074 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
W20260619 06:56:52.551892 7072 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
I20260619 06:56:52.552045 7072 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5949f652fc46df9facc33661d6d4ec; no voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e
I20260619 06:56:52.552351 7458 raft_consensus.cc:2726] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:52.862891 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:52.862555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:50957)
0619 06:56:52.862739 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:52.862807 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:56:53.015820 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:53.015216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:33853)
0619 06:56:53.015610 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:53.015675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":112,"threads_started":1}
I20260619 06:56:53.222436 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:53.221951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:41825)
0619 06:56:53.222246 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:53.222319 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260619 06:56:53.864315 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:53.863814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:55669)
0619 06:56:53.864142 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:53.864212 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260619 06:56:54.017472 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:54.016885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:33383)
0619 06:56:54.017251 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:54.017332 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260619 06:56:54.224195 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:54.223579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:44853)
0619 06:56:54.223933 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:54.224014 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260619 06:56:54.608793 7468 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:54.608999 7468 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:54.609558 7468 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
W20260619 06:56:54.610692 7072 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
W20260619 06:56:54.610879 7074 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
I20260619 06:56:54.611032 7074 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5949f652fc46df9facc33661d6d4ec; no voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e
I20260619 06:56:54.611274 7468 raft_consensus.cc:2726] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:54.865778 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:54.865221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:59831)
0619 06:56:54.865553 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:54.865617 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":133,"threads_started":1}
I20260619 06:56:55.019178 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:55.018542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:37275)
0619 06:56:55.018938 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:55.019032 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260619 06:56:55.225804 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:55.225172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:52009)
0619 06:56:55.225598 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:55.225681 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260619 06:56:55.867347 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:55.866745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:50141)
0619 06:56:55.867094 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:55.867158 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260619 06:56:56.020718 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:56.020215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:50447)
0619 06:56:56.020531 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:56.020604 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260619 06:56:56.227443 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:56.226832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:55589)
0619 06:56:56.227206 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:56.227330 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260619 06:56:56.699398 7475 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:56.699564 7475 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:56.700157 7475 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
W20260619 06:56:56.701128 7074 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
W20260619 06:56:56.701231 7072 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
I20260619 06:56:56.701297 7072 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5949f652fc46df9facc33661d6d4ec; no voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e
I20260619 06:56:56.701462 7475 raft_consensus.cc:2726] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:56.868680 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:56.868173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:43965)
0619 06:56:56.868469 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:56.868549 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260619 06:56:57.022493 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:57.021779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:44787)
0619 06:56:57.022248 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:57.022343 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":138,"threads_started":1}
I20260619 06:56:57.228884 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:57.228316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:59695)
0619 06:56:57.228675 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:57.228758 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260619 06:56:57.870200 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:57.869674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:41967)
0619 06:56:57.870004 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:57.870077 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
I20260619 06:56:58.023969 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:58.023381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:38593)
0619 06:56:58.023763 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:58.023838 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":89,"threads_started":1}
I20260619 06:56:58.230700 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:58.229948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:39443)
0619 06:56:58.230481 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:58.230561 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":106,"threads_started":1}
I20260619 06:56:58.867936 7482 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:56:58.868089 7482 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:56:58.868628 7482 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
W20260619 06:56:58.869701 7072 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
W20260619 06:56:58.869825 7074 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
I20260619 06:56:58.869969 7074 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5949f652fc46df9facc33661d6d4ec; no voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e
I20260619 06:56:58.870214 7482 raft_consensus.cc:2726] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:56:58.871428 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:58.870978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:44169)
0619 06:56:58.871235 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:58.871342 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
I20260619 06:56:59.025667 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:59.024864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:51809)
0619 06:56:59.025209 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:59.025550 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260619 06:56:59.232169 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:59.231608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:60543)
0619 06:56:59.231954 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:59.232039 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":81,"threads_started":1}
I20260619 06:56:59.872714 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:56:59.872181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:44201)
0619 06:56:59.872520 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 06:56:59.872603 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260619 06:57:00.027029 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:00.026493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:46639)
0619 06:57:00.026827 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:00.026904 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":151,"threads_started":1}
I20260619 06:57:00.233887 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:00.233294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:43257)
0619 06:57:00.233650 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:00.233720 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":140,"threads_started":1}
I20260619 06:57:00.874287 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:00.873786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:39245)
0619 06:57:00.874107 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:00.874183 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260619 06:57:01.028890 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:01.028301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:54201)
0619 06:57:01.028691 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:01.028776 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260619 06:57:01.235538 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:01.234900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:55599)
0619 06:57:01.235306 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:01.235400 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
I20260619 06:57:01.875794 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:01.875226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:41435)
0619 06:57:01.875593 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:01.875666 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":80,"threads_started":1}
I20260619 06:57:02.030669 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:02.030047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:53373)
0619 06:57:02.030455 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:02.030537 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260619 06:57:02.237197 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:02.236621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:59535)
0619 06:57:02.236973 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:02.237067 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":89,"threads_started":1}
I20260619 06:57:02.564472 7495 raft_consensus.cc:482] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:02.564642 7495 raft_consensus.cc:504] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f966db3513d4d5fb735f1e598d20c9e" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 38605 } } peers { permanent_uuid: "67c03fde360b4ba4a226fd763aef14d3" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 32783 } } peers { permanent_uuid: "cc5949f652fc46df9facc33661d6d4ec" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46851 } }
I20260619 06:57:02.565179 7495 leader_election.cc:288] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605), 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783)
W20260619 06:57:02.566202 7072 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f966db3513d4d5fb735f1e598d20c9e (127.22.113.3:38605): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
W20260619 06:57:02.566342 7074 leader_election.cc:341] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67c03fde360b4ba4a226fd763aef14d3 (127.22.113.2:32783): Not found: Tablet not found: 150b331df7c1452f96de272268ddb7ee
I20260619 06:57:02.566470 7074 leader_election.cc:302] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5949f652fc46df9facc33661d6d4ec; no voters: 67c03fde360b4ba4a226fd763aef14d3, 6f966db3513d4d5fb735f1e598d20c9e
I20260619 06:57:02.566702 7495 raft_consensus.cc:2726] T 150b331df7c1452f96de272268ddb7ee P cc5949f652fc46df9facc33661d6d4ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:57:02.877396 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:02.876829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:43397)
0619 06:57:02.877175 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:02.877259 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":90,"threads_started":1}
I20260619 06:57:03.032203 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:03.031623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:56267)
0619 06:57:03.031990 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:03.032068 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
I20260619 06:57:03.238731 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:03.238187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:41023)
0619 06:57:03.238538 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:03.238603 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
I20260619 06:57:03.878846 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:03.878301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:35897)
0619 06:57:03.878644 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:03.878717 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260619 06:57:04.033969 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:04.033314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:44363)
0619 06:57:04.033734 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:04.033852 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":127,"threads_started":1}
I20260619 06:57:04.240311 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:04.239749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:47785)
0619 06:57:04.240129 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:04.240196 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260619 06:57:04.880384 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:04.879800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:58427)
0619 06:57:04.880157 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:04.880243 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260619 06:57:05.035648 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:05.034909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:33721)
0619 06:57:05.035418 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:05.035512 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260619 06:57:05.241905 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:05.241322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:33101)
0619 06:57:05.241663 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:05.241750 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260619 06:57:05.882650 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:05.882032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.1:58857)
0619 06:57:05.882428 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:05.882518 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260619 06:57:06.037556 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:06.036975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.2:40955)
0619 06:57:06.037339 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:06.037423 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260619 06:57:06.230494 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7015
I20260619 06:57:06.237660 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7189
I20260619 06:57:06.243705 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:06.243097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40579 (local address 127.22.113.3:60643)
0619 06:57:06.243490 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:06.243566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.62:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260619 06:57:06.244589 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7322
2026-06-19T06:57:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59576 ms)
[----------] 30 tests from SecurityITest (249715 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:06 dist-test-slave-g8zg krb5kdc[7510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:06 dist-test-slave-g8zg krb5kdc[7510](info): set up 2 sockets
Jun 19 06:57:06 dist-test-slave-g8zg krb5kdc[7510](info): commencing operation
krb5kdc: starting...
W20260619 06:57:07.299463 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:07 dist-test-slave-g8zg krb5kdc[7510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:07.438978 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36337
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:07.556748 7523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:07.557025 7523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:07.557065 7523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:07.560922 7523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:07.560982 7523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:07.561002 7523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:07.561018 7523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:07.561034 7523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:07.566160 7523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:36337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:07.567188 7523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:07.568236 7523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:07.578310 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:07.578382 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:07.578454 7523 server_base.cc:1034] running on GCE node
W20260619 06:57:07.578510 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:07.579577 7523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:07.580899 7523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:07.582095 7523 hybrid_clock.cc:648] HybridClock initialized: now 1781852227582058 us; error 49 us; skew 500 ppm
Jun 19 06:57:07 dist-test-slave-g8zg krb5kdc[7510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:07.585439 7523 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:07.586602 7523 webserver.cc:460] Webserver started at http://127.22.113.62:43361/ using document root <none> and password file <none>
I20260619 06:57:07.587363 7523 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:07.587472 7523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:07.587749 7523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:07.590190 7523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "e8c31fedf9254dfe8066e33ad1059c21"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "b04fd73783093f1e6552dc2bc27c6eca"
server_key_iv: "0c5afc00df94e0e6cd90e4b8e6e02706"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.590830 7523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "e8c31fedf9254dfe8066e33ad1059c21"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "b04fd73783093f1e6552dc2bc27c6eca"
server_key_iv: "0c5afc00df94e0e6cd90e4b8e6e02706"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.595427 7523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:57:07.598646 7538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:07.600338 7523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:07.600541 7523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "e8c31fedf9254dfe8066e33ad1059c21"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "b04fd73783093f1e6552dc2bc27c6eca"
server_key_iv: "0c5afc00df94e0e6cd90e4b8e6e02706"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.600698 7523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:07.617624 7523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:07.621631 7523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:07.621917 7523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:07.631824 7523 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:36337
I20260619 06:57:07.631856 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:36337 every 8 connection(s)
I20260619 06:57:07.633111 7523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:07.636492 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:07.637034 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7523
I20260619 06:57:07.637187 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:07.637612 22980 external_mini_cluster.cc:1428] Setting key 9a65fd1da92315344f78f601e85644e0
I20260619 06:57:07.643884 7591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: Bootstrap starting.
I20260619 06:57:07.646608 7591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:07.647519 7591 log.cc:824] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:07 dist-test-slave-g8zg krb5kdc[7510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:07.650534 7591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: No bootstrap required, opened a new log
I20260619 06:57:07.655364 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:07.639522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.22.113.62:36337)
0619 06:57:07.640105 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:07.640127 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:07.640181 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:57:07.641080 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:07.641104 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:07.641542 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:07.641940 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:07.642719 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.644223 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.645271 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.645989 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.646177 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:07.649578 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:07.649585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:07.649601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:07.649631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:07.651926 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:07.652461 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:07.652467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:07.652472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:07.652622 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:07.652935 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:07.652939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:07.652941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:07.653552 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:07.653742 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:07.654037 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:57:07.654356 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":167,"threads_started":1}
I20260619 06:57:07.655426 7591 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } }
I20260619 06:57:07.655686 7591 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:07.655748 7591 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c31fedf9254dfe8066e33ad1059c21, State: Initialized, Role: FOLLOWER
I20260619 06:57:07.656235 7591 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } }
I20260619 06:57:07.656395 7591 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:07.656454 7591 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:07.656558 7591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:07.657709 7591 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } }
I20260619 06:57:07.658121 7591 leader_election.cc:302] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c31fedf9254dfe8066e33ad1059c21; no voters:
I20260619 06:57:07.658434 7591 leader_election.cc:288] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:07.658604 7596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:07.658907 7596 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [term 1 LEADER]: Becoming Leader. State: Replica: e8c31fedf9254dfe8066e33ad1059c21, State: Running, Role: LEADER
I20260619 06:57:07.659343 7596 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } }
I20260619 06:57:07.659930 7591 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:07.661337 7597 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c31fedf9254dfe8066e33ad1059c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } } }
I20260619 06:57:07.661500 7597 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:07.661814 7598 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c31fedf9254dfe8066e33ad1059c21. Latest consensus state: current_term: 1 leader_uuid: "e8c31fedf9254dfe8066e33ad1059c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c31fedf9254dfe8066e33ad1059c21" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 36337 } } }
I20260619 06:57:07.661989 7598 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21 [sys.catalog]: This master's current role is: LEADER
W20260619 06:57:07.665958 7611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:07.666074 7611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:07.666150 7606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:07.666913 7606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:07.674525 7606 catalog_manager.cc:1347] Generated new cluster ID: 728c58a3bb6349b8ac8bc7b52b7a074b
I20260619 06:57:07.674613 7606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:07.688515 7606 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:07.689684 7606 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:07.707069 7606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: Generated new TSK 0
I20260619 06:57:07.708160 7606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:57:07.751748 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:07.883379 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:07.883689 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:07.883736 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:07.887825 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:07.887880 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:07.887940 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:07.893251 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:07.894447 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:07.895660 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:07.906912 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:07.906975 7617 server_base.cc:1034] running on GCE node
W20260619 06:57:07.906937 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:07.906989 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:07.907816 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:07.908584 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:07.909827 7617 hybrid_clock.cc:648] HybridClock initialized: now 1781852227909808 us; error 35 us; skew 500 ppm
Jun 19 06:57:07 dist-test-slave-g8zg krb5kdc[7510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:07.913100 7617 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:57:07.914064 7617 webserver.cc:460] Webserver started at http://127.22.113.1:38813/ using document root <none> and password file <none>
I20260619 06:57:07.914718 7617 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:07.914784 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:07.915015 7617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:07.917496 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "cde0cbe9dd854bf1b5420f4f043bc84a"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "8a4ba3522200c99b99fbb29cc73c4eb6"
server_key_iv: "78171494ad7195bedde7d591a6b8a4d4"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.918092 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "cde0cbe9dd854bf1b5420f4f043bc84a"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "8a4ba3522200c99b99fbb29cc73c4eb6"
server_key_iv: "78171494ad7195bedde7d591a6b8a4d4"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.922223 7617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:07.925009 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:07.926482 7617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:07.926656 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "cde0cbe9dd854bf1b5420f4f043bc84a"
format_stamp: "Formatted at 2026-06-19 06:57:07 on dist-test-slave-g8zg"
server_key: "8a4ba3522200c99b99fbb29cc73c4eb6"
server_key_iv: "78171494ad7195bedde7d591a6b8a4d4"
server_key_version: "encryptionkey@0"
I20260619 06:57:07.926769 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:07.940394 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:07.943665 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:07.943876 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:07.944516 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:07.945386 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:07.945436 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:07.945513 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:07.945554 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:07.955998 7617 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:36899
I20260619 06:57:07.956096 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:36899 every 8 connection(s)
I20260619 06:57:07.957002 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:07.962821 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7617
I20260619 06:57:07.963008 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:07.963464 22980 external_mini_cluster.cc:1428] Setting key a0618978082ae3b1b3d198b6ed16649c
Jun 19 06:57:07 dist-test-slave-g8zg krb5kdc[7510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:07.974331 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:07.959322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:37303 (local address 127.22.113.62:36337)
0619 06:57:07.959496 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:07.959500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:07.960544 (+ 1044us) server_negotiation.cc:407] Connection header received
0619 06:57:07.962389 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:07.962393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:07.962461 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:07.962655 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:07.964322 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.964877 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.966084 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.966324 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.966439 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:07.969756 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:07.969763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:07.969770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:07.969808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:07.971731 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:07.972395 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:07.972399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:07.972401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:07.972519 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:07.972897 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:07.972900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:07.972902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:07.973171 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:07.973308 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:07.973924 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 06:57:07.974093 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:57:07.975520 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:07.959699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36337 (local address 127.22.113.1:37303)
0619 06:57:07.960221 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:07.960342 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:07.962108 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:07.962896 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:07.962904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:07.963467 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:07.964129 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:07.964154 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.965023 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.965026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:07.965941 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:07.965950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:07.966484 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:07.966488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:07.966667 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:07.967473 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:07.967500 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:07.969525 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:07.971889 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:07.971896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:07.971912 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:07.972255 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:07.972625 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:07.972628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:07.972629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:07.972777 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:07.973301 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:07.973306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:07.973624 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:57:07.973803 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:57:07.974269 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260619 06:57:07.977283 7746 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:57:07.977618 7746 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:07.978302 7746 heartbeater.cc:511] Master 127.22.113.62:36337 requested a full tablet report, sending...
I20260619 06:57:07.980314 7555 ts_manager.cc:194] Registered new tserver with Master: cde0cbe9dd854bf1b5420f4f043bc84a (127.22.113.1:36899)
I20260619 06:57:07.982030 7555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:37303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:57:07.995069 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:08.112341 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:08.112612 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:08.112654 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:08.116722 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:08.116786 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:08.116847 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:08.122203 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:08.123291 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:08.124289 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:08.134821 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:08.134845 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:08.134846 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:08.135270 7751 server_base.cc:1034] running on GCE node
I20260619 06:57:08.135800 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:08.136504 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:08.137714 7751 hybrid_clock.cc:648] HybridClock initialized: now 1781852228137689 us; error 41 us; skew 500 ppm
Jun 19 06:57:08 dist-test-slave-g8zg krb5kdc[7510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852228, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:08.140863 7751 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:57:08.141930 7751 webserver.cc:460] Webserver started at http://127.22.113.2:38273/ using document root <none> and password file <none>
I20260619 06:57:08.142568 7751 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:08.142668 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:08.142920 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:08.145268 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "637c84c7deb247dc80938e12931c1d76"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "6f89e52181bfd526674f4fb554ad345b"
server_key_iv: "a97673b84ed6f3988406bf161e9cb8b3"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.146219 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "637c84c7deb247dc80938e12931c1d76"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "6f89e52181bfd526674f4fb554ad345b"
server_key_iv: "a97673b84ed6f3988406bf161e9cb8b3"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.150363 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:57:08.153299 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.154680 7751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:08.154829 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "637c84c7deb247dc80938e12931c1d76"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "6f89e52181bfd526674f4fb554ad345b"
server_key_iv: "a97673b84ed6f3988406bf161e9cb8b3"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.154943 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:08.176046 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:08.179062 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:08.179313 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:08.179910 7751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:08.180740 7751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:08.180819 7751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.180889 7751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:08.180932 7751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.190498 7751 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:45799
I20260619 06:57:08.190544 7879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:45799 every 8 connection(s)
I20260619 06:57:08.191491 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:08.192589 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7751
I20260619 06:57:08.192721 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:08.193043 22980 external_mini_cluster.cc:1428] Setting key 45a3cf0bab95ff0c4d65659f7e871e71
Jun 19 06:57:08 dist-test-slave-g8zg krb5kdc[7510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852228, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:57:08.207715 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.193858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:48265 (local address 127.22.113.62:36337)
0619 06:57:08.194055 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.194060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.194903 (+ 843us) server_negotiation.cc:407] Connection header received
0619 06:57:08.196380 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.196386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.196454 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.196558 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:08.197849 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.198400 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.199635 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.199851 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.199972 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.203099 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:08.203104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:08.203108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:08.203152 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:08.205034 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.205688 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.205692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.205693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.205774 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.206193 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.206198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.206200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.206471 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:08.206561 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.207358 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.207536 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:57:08.208570 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.194237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36337 (local address 127.22.113.2:48265)
0619 06:57:08.194720 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.194771 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.196149 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.196760 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.196768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.197125 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:08.197656 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.197670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.198542 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.198545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.199509 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.199520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.200004 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.200008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.200183 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.200939 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:08.200955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:08.202825 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:08.205160 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:08.205165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:08.205178 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:08.205470 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:08.205873 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:08.205875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:08.205877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:08.206103 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:08.206648 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:08.206653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:08.206994 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.207243 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.207620 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260619 06:57:08.210026 7880 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36337
I20260619 06:57:08.210387 7880 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:08.211077 7880 heartbeater.cc:511] Master 127.22.113.62:36337 requested a full tablet report, sending...
I20260619 06:57:08.212450 7555 ts_manager.cc:194] Registered new tserver with Master: 637c84c7deb247dc80938e12931c1d76 (127.22.113.2:45799)
I20260619 06:57:08.213409 7555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:48265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:57:08.228511 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:08.359745 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:08.360066 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:08.360153 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:08.364755 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:08.364845 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:08.364954 7885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:08.370887 7885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:08.372303 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:08.373569 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:08.384922 7893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:08.384964 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:08.385120 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:08.385437 7885 server_base.cc:1034] running on GCE node
I20260619 06:57:08.386022 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:08.386759 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:08.387949 7885 hybrid_clock.cc:648] HybridClock initialized: now 1781852228387930 us; error 32 us; skew 500 ppm
Jun 19 06:57:08 dist-test-slave-g8zg krb5kdc[7510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852228, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:08.391024 7885 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:57:08.392072 7885 webserver.cc:460] Webserver started at http://127.22.113.3:34475/ using document root <none> and password file <none>
I20260619 06:57:08.392714 7885 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:08.392807 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:08.393044 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:08.395138 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "ca83000c3cc5444eb7b685f885969dcf"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "81e1c2654db424785b651a4c4d48b734"
server_key_iv: "c256a211487566728634b726c5911bf5"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.395813 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "ca83000c3cc5444eb7b685f885969dcf"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "81e1c2654db424785b651a4c4d48b734"
server_key_iv: "c256a211487566728634b726c5911bf5"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.400311 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:57:08.403120 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.404702 7885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:08.404851 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "ca83000c3cc5444eb7b685f885969dcf"
format_stamp: "Formatted at 2026-06-19 06:57:08 on dist-test-slave-g8zg"
server_key: "81e1c2654db424785b651a4c4d48b734"
server_key_iv: "c256a211487566728634b726c5911bf5"
server_key_version: "encryptionkey@0"
I20260619 06:57:08.404968 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:08.414491 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:08.417377 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:08.417568 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:08.418134 7885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:08.418938 7885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:08.419010 7885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.419075 7885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:08.419135 7885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:08.429159 7885 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40077
I20260619 06:57:08.429237 8013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40077 every 8 connection(s)
I20260619 06:57:08.430203 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:08.436784 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 7885
I20260619 06:57:08.436997 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:08.437351 22980 external_mini_cluster.cc:1428] Setting key abcbe84f679e0e52714f306667629d1e
Jun 19 06:57:08 dist-test-slave-g8zg krb5kdc[7510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852228, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:08.446084 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.432549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:58159 (local address 127.22.113.62:36337)
0619 06:57:08.432725 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.432729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.433625 (+ 896us) server_negotiation.cc:407] Connection header received
0619 06:57:08.435164 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.435168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.435231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.435373 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:08.436857 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.437574 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.439240 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.439578 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.439707 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.442513 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:08.442516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:08.442519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:08.442544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:08.444145 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.444664 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.444667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.444668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.444745 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.445052 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.445054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.445055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.445243 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:08.445318 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.445850 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.445947 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:57:08.447065 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.432908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:36337 (local address 127.22.113.3:58159)
0619 06:57:08.433444 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.433493 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.434928 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.435579 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.435587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.436007 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:08.436656 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.436678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.437728 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.437734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.439109 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.439117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.439771 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.439776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.439986 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.440708 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:08.440726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:08.442304 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:08.444253 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:08.444258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:08.444271 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:08.444560 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:08.444842 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:08.444844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:08.444846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:08.444971 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:08.445342 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:08.445347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:08.445621 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.445794 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.446072 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260619 06:57:08.448410 8014 heartbeater.cc:347] Connected to a master server at 127.22.113.62:36337
I20260619 06:57:08.448689 8014 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:08.449250 8014 heartbeater.cc:511] Master 127.22.113.62:36337 requested a full tablet report, sending...
I20260619 06:57:08.450446 7555 ts_manager.cc:194] Registered new tserver with Master: ca83000c3cc5444eb7b685f885969dcf (127.22.113.3:40077)
I20260619 06:57:08.451172 7555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:58159
I20260619 06:57:08.451997 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:08.462267 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.454352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.22.113.62:36337)
0619 06:57:08.454503 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.454507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.454755 (+ 248us) server_negotiation.cc:407] Connection header received
0619 06:57:08.454860 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.454863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.454909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.454983 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:08.455828 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.456384 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.457282 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.457508 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.457586 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.459008 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:08.459012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:08.459015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:08.459052 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:08.460455 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.460953 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.460957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.460958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.461046 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:08.461312 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:08.461316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:08.461318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:08.461606 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:08.461742 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.461962 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.462098 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:57:08.465342 7555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:08.468554 7555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:08.484652 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.478263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:40077 (local address 127.0.0.1:34340)
0619 06:57:08.479769 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.479782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.479886 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.480585 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.480594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.480611 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:08.480750 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.480757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.481976 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.481979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.483122 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.483132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.484057 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.484061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.484188 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.484233 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.484352 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.484481 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":922,"spinlock_wait_cycles":768,"thread_start_us":153,"threads_started":1}
I20260619 06:57:08.484712 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.479111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.22.113.3:40077)
0619 06:57:08.479580 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.479590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.479861 (+ 271us) server_negotiation.cc:407] Connection header received
0619 06:57:08.480088 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.480095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.480299 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.480502 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:08.480917 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.481801 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.483274 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.483923 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.484038 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.484208 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.484409 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.484525 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":175,"threads_started":1}
I20260619 06:57:08.485600 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.478410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.22.113.2:45799)
0619 06:57:08.479008 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.479022 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.479773 (+ 751us) server_negotiation.cc:407] Connection header received
0619 06:57:08.479990 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.479997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.480224 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.480381 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:08.481022 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.482109 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.483905 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.484752 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.484836 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.484962 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.485338 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.485445 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":146,"threads_started":1}
I20260619 06:57:08.486027 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.478263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45799 (local address 127.0.0.1:59894)
0619 06:57:08.479618 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.479669 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.479842 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.480688 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.480692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.480715 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:08.480811 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.480815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.482264 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.482267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.483762 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.483771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.485082 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.485085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.485176 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.485198 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.485812 (+ 614us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.485890 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":178,"thread_start_us":124,"threads_started":1}
I20260619 06:57:08.486440 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.478470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36899 (local address 127.0.0.1:33840)
0619 06:57:08.479621 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.479676 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.479886 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.480410 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.480420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.480449 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:08.480622 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.480633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.482210 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.482215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.483759 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.483771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.485930 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.485933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.486048 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.486072 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.486144 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.486245 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":63,"spinlock_wait_cycles":1280,"thread_start_us":96,"threads_started":1}
I20260619 06:57:08.486986 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.478886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.22.113.1:36899)
0619 06:57:08.479345 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.479352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.479775 (+ 423us) server_negotiation.cc:407] Connection header received
0619 06:57:08.480059 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.480065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.480223 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.480381 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:08.480765 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.482044 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.483923 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.485349 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.485720 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.485861 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.486670 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.486805 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260619 06:57:08.488304 7948 tablet_service.cc:1467] Processing CreateTablet for tablet f65850fcc540457d8662ab2fffdd1a45 (DEFAULT_TABLE table=test-table [id=ca8340d0a26e427e85dc76283e9db4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:08.489586 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65850fcc540457d8662ab2fffdd1a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:08.489467 7814 tablet_service.cc:1467] Processing CreateTablet for tablet f65850fcc540457d8662ab2fffdd1a45 (DEFAULT_TABLE table=test-table [id=ca8340d0a26e427e85dc76283e9db4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:08.490695 7814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65850fcc540457d8662ab2fffdd1a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:08.490439 7680 tablet_service.cc:1467] Processing CreateTablet for tablet f65850fcc540457d8662ab2fffdd1a45 (DEFAULT_TABLE table=test-table [id=ca8340d0a26e427e85dc76283e9db4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:08.491676 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65850fcc540457d8662ab2fffdd1a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:08.497267 8031 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Bootstrap starting.
I20260619 06:57:08.498513 8032 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Bootstrap starting.
I20260619 06:57:08.499531 8033 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Bootstrap starting.
I20260619 06:57:08.499648 8031 tablet_bootstrap.cc:654] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:08.500669 8032 tablet_bootstrap.cc:654] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:08.501004 8031 log.cc:824] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:08.501421 8032 log.cc:824] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:08.501646 8033 tablet_bootstrap.cc:654] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:08.503542 8032 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: No bootstrap required, opened a new log
I20260619 06:57:08.503542 8031 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: No bootstrap required, opened a new log
I20260619 06:57:08.503744 8032 ts_tablet_manager.cc:1397] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 06:57:08.503892 8031 ts_tablet_manager.cc:1397] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:57:08.504180 8033 log.cc:824] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:08.506767 8033 tablet_bootstrap.cc:492] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: No bootstrap required, opened a new log
I20260619 06:57:08.507002 8033 ts_tablet_manager.cc:1397] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260619 06:57:08.509027 8032 raft_consensus.cc:348] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.509433 8032 raft_consensus.cc:374] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:08.509495 8032 raft_consensus.cc:729] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde0cbe9dd854bf1b5420f4f043bc84a, State: Initialized, Role: FOLLOWER
I20260619 06:57:08.510010 8032 consensus_queue.cc:260] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.510506 8031 raft_consensus.cc:348] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.510804 8031 raft_consensus.cc:374] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:08.510892 8031 raft_consensus.cc:729] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca83000c3cc5444eb7b685f885969dcf, State: Initialized, Role: FOLLOWER
I20260619 06:57:08.511058 8032 ts_tablet_manager.cc:1428] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260619 06:57:08.511453 7746 heartbeater.cc:503] Master 127.22.113.62:36337 was elected leader, sending a full tablet report...
I20260619 06:57:08.511420 8031 consensus_queue.cc:260] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.512323 8031 ts_tablet_manager.cc:1428] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260619 06:57:08.512120 8033 raft_consensus.cc:348] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.512429 8033 raft_consensus.cc:374] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:08.512495 8033 raft_consensus.cc:729] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637c84c7deb247dc80938e12931c1d76, State: Initialized, Role: FOLLOWER
I20260619 06:57:08.512495 8014 heartbeater.cc:503] Master 127.22.113.62:36337 was elected leader, sending a full tablet report...
I20260619 06:57:08.512984 8033 consensus_queue.cc:260] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.514557 7880 heartbeater.cc:503] Master 127.22.113.62:36337 was elected leader, sending a full tablet report...
I20260619 06:57:08.515372 8033 ts_tablet_manager.cc:1428] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
W20260619 06:57:08.681463 8015 tablet.cc:2369] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:57:08.692667 7881 tablet.cc:2369] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:08.699873 8039 raft_consensus.cc:482] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:08.700073 8039 raft_consensus.cc:504] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.701270 8039 leader_election.cc:288] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca83000c3cc5444eb7b685f885969dcf (127.22.113.3:40077), cde0cbe9dd854bf1b5420f4f043bc84a (127.22.113.1:36899)
I20260619 06:57:08.705336 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.701661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:40077 (local address 127.22.113.2:60025)
0619 06:57:08.701922 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.701941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.702066 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.702389 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.702393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.702417 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:08.702548 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.702555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.703320 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.703326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.704192 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.704204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.704875 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.704879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.704996 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.705024 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.705114 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.705198 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260619 06:57:08.705336 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.701761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:60025 (local address 127.22.113.3:40077)
0619 06:57:08.701918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.701923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.702026 (+ 103us) server_negotiation.cc:407] Connection header received
0619 06:57:08.702199 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.702201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.702256 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.702358 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:08.702664 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.703186 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.704314 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.704753 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.704888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.704946 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.705129 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.705220 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:57:08.706101 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.701667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36899 (local address 127.22.113.2:41345)
0619 06:57:08.702162 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.702177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:08.702311 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:08.702586 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:08.702589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:08.702606 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:08.702757 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.702763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.703607 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.703613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.704638 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:08.704648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.705644 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.705648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:08.705763 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.705788 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:57:08.705867 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:57:08.705942 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":72,"thread_start_us":111,"threads_started":1}
I20260619 06:57:08.706164 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.701792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:41345 (local address 127.22.113.1:36899)
0619 06:57:08.701957 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.701961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.702290 (+ 329us) server_negotiation.cc:407] Connection header received
0619 06:57:08.702429 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.702432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.702482 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.702573 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:08.702867 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.703495 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.704760 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.705305 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.705444 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.705544 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.705892 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.706041 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:57:08.706292 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65850fcc540457d8662ab2fffdd1a45" candidate_uuid: "637c84c7deb247dc80938e12931c1d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca83000c3cc5444eb7b685f885969dcf" is_pre_election: true
I20260619 06:57:08.706604 7968 raft_consensus.cc:2454] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 637c84c7deb247dc80938e12931c1d76 in term 0.
I20260619 06:57:08.707166 7770 leader_election.cc:302] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637c84c7deb247dc80938e12931c1d76, ca83000c3cc5444eb7b685f885969dcf; no voters:
I20260619 06:57:08.707211 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65850fcc540457d8662ab2fffdd1a45" candidate_uuid: "637c84c7deb247dc80938e12931c1d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" is_pre_election: true
I20260619 06:57:08.707517 8039 raft_consensus.cc:2781] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:08.707635 8039 raft_consensus.cc:482] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:08.707639 7700 raft_consensus.cc:2454] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 637c84c7deb247dc80938e12931c1d76 in term 0.
I20260619 06:57:08.707690 8039 raft_consensus.cc:3037] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 0 FOLLOWER]: Advancing to term 1
W20260619 06:57:08.708379 7747 tablet.cc:2369] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:08.709106 8039 raft_consensus.cc:504] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.709544 8039 leader_election.cc:288] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [CANDIDATE]: Term 1 election: Requested vote from peers ca83000c3cc5444eb7b685f885969dcf (127.22.113.3:40077), cde0cbe9dd854bf1b5420f4f043bc84a (127.22.113.1:36899)
I20260619 06:57:08.709961 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65850fcc540457d8662ab2fffdd1a45" candidate_uuid: "637c84c7deb247dc80938e12931c1d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca83000c3cc5444eb7b685f885969dcf"
I20260619 06:57:08.709981 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65850fcc540457d8662ab2fffdd1a45" candidate_uuid: "637c84c7deb247dc80938e12931c1d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a"
I20260619 06:57:08.710150 7700 raft_consensus.cc:3037] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:08.710093 7968 raft_consensus.cc:3037] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:08.711398 7700 raft_consensus.cc:2454] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637c84c7deb247dc80938e12931c1d76 in term 1.
I20260619 06:57:08.711401 7968 raft_consensus.cc:2454] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637c84c7deb247dc80938e12931c1d76 in term 1.
I20260619 06:57:08.711736 7767 leader_election.cc:302] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637c84c7deb247dc80938e12931c1d76, cde0cbe9dd854bf1b5420f4f043bc84a; no voters:
I20260619 06:57:08.711941 8039 raft_consensus.cc:2781] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:08.712230 8039 raft_consensus.cc:686] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 1 LEADER]: Becoming Leader. State: Replica: 637c84c7deb247dc80938e12931c1d76, State: Running, Role: LEADER
I20260619 06:57:08.712664 8039 consensus_queue.cc:237] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } }
I20260619 06:57:08.716355 7555 catalog_manager.cc:5495] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 637c84c7deb247dc80938e12931c1d76 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "637c84c7deb247dc80938e12931c1d76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637c84c7deb247dc80938e12931c1d76" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20260619 06:57:08.785840 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.782125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.22.113.2:45799)
0619 06:57:08.782304 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:08.782308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:08.782327 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:57:08.782426 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:08.782430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:08.782499 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:08.782636 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:08.782895 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.783488 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.784349 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:08.784575 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:08.784691 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:08.785004 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:08.785447 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:08.785580 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:08.785624 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 06:57:08.785681 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:57:08.792954 7700 raft_consensus.cc:1261] T f65850fcc540457d8662ab2fffdd1a45 P cde0cbe9dd854bf1b5420f4f043bc84a [term 1 FOLLOWER]: Refusing update from remote peer 637c84c7deb247dc80938e12931c1d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:08.792982 7968 raft_consensus.cc:1261] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 1 FOLLOWER]: Refusing update from remote peer 637c84c7deb247dc80938e12931c1d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:08.793784 8039 consensus_queue.cc:1035] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde0cbe9dd854bf1b5420f4f043bc84a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:08.794018 8041 consensus_queue.cc:1035] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca83000c3cc5444eb7b685f885969dcf" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:08.812917 7553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:08.813139 7553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:08.815830 7553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c31fedf9254dfe8066e33ad1059c21: Sending DeleteTablet for 3 replicas of tablet f65850fcc540457d8662ab2fffdd1a45
I20260619 06:57:08.816720 7948 tablet_service.cc:1514] Processing DeleteTablet for tablet f65850fcc540457d8662ab2fffdd1a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:08 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:34340
I20260619 06:57:08.816825 7814 tablet_service.cc:1514] Processing DeleteTablet for tablet f65850fcc540457d8662ab2fffdd1a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:08 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59894
I20260619 06:57:08.817308 7680 tablet_service.cc:1514] Processing DeleteTablet for tablet f65850fcc540457d8662ab2fffdd1a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:08 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:33840
I20260619 06:57:08.817515 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7617
I20260619 06:57:08.818284 8055 tablet_replica.cc:330] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: stopping tablet replica
I20260619 06:57:08.818754 8055 raft_consensus.cc:2229] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:08.819131 8055 raft_consensus.cc:2258] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:08.820909 8055 ts_tablet_manager.cc:1905] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:08.823526 8055 ts_tablet_manager.cc:1918] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:08.823652 8055 log.cc:1196] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/f65850fcc540457d8662ab2fffdd1a45
I20260619 06:57:08.823900 8054 tablet_replica.cc:330] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76: stopping tablet replica
I20260619 06:57:08.824056 8055 ts_tablet_manager.cc:1939] T f65850fcc540457d8662ab2fffdd1a45 P ca83000c3cc5444eb7b685f885969dcf: Deleting consensus metadata
I20260619 06:57:08.824476 8054 raft_consensus.cc:2229] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:57:08.825037 8054 raft_consensus.cc:2258] T f65850fcc540457d8662ab2fffdd1a45 P 637c84c7deb247dc80938e12931c1d76 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:57:08.824975 7539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:36899 (error 108)
I20260619 06:57:08.825654 7542 catalog_manager.cc:4841] TS ca83000c3cc5444eb7b685f885969dcf (127.22.113.3:40077): tablet f65850fcc540457d8662ab2fffdd1a45 (table test-table [id=ca8340d0a26e427e85dc76283e9db4f1]) successfully deleted
I20260619 06:57:08.825858 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7751
I20260619 06:57:08.827890 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.827536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36899 (local address 127.0.0.1:33846)
0619 06:57:08.827735 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.827802 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260619 06:57:08.828177 7539 catalog_manager.cc:4584] TS cde0cbe9dd854bf1b5420f4f043bc84a (127.22.113.1:36899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f65850fcc540457d8662ab2fffdd1a45: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36899: connect: Connection refused (error 111)
I20260619 06:57:08.833751 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7885
I20260619 06:57:08.836045 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:08.833886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45799 (local address 127.0.0.1:59908)
0619 06:57:08.835898 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:08.835956 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.2:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1902}
I20260619 06:57:08.840202 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 7523
2026-06-19T06:57:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2598 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T06:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:08Z Disabled control of system clock
I20260619 06:57:08.879491 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:37091
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:37091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:08.999704 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:08.999986 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:09.000023 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:09.004102 8061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:09.004163 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:09.004179 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:09.004195 8061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:09.004211 8061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:09.009258 8061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:37091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:37091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:09.010406 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:09.011817 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:09.021216 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.021320 8061 server_base.cc:1034] running on GCE node
W20260619 06:57:09.021221 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:09.021221 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.022099 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:09.023223 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:09.024442 8061 hybrid_clock.cc:648] HybridClock initialized: now 1781852229024406 us; error 55 us; skew 500 ppm
I20260619 06:57:09.026369 8061 webserver.cc:460] Webserver started at http://127.22.113.62:34571/ using document root <none> and password file <none>
I20260619 06:57:09.026952 8061 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:09.027017 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:09.027451 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:09.029569 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "a5c166be8582c1b62fed20e493a54577"
server_key_iv: "0ebf9276c97b5551be958b8e364c2a8c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.030143 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "a5c166be8582c1b62fed20e493a54577"
server_key_iv: "0ebf9276c97b5551be958b8e364c2a8c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.034058 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:09.036702 8075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.037995 8061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:09.038141 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "a5c166be8582c1b62fed20e493a54577"
server_key_iv: "0ebf9276c97b5551be958b8e364c2a8c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.038246 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:09.085419 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:09.086155 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:09.086364 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:09.094363 8061 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:37091
I20260619 06:57:09.094444 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:37091 every 8 connection(s)
I20260619 06:57:09.095520 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:09.097096 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8061
I20260619 06:57:09.097203 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:09.097514 22980 external_mini_cluster.cc:1428] Setting key 8feb4c94afa8eb9c05c70aceb98f6f5d
I20260619 06:57:09.099799 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:09.107862 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: Bootstrap starting.
I20260619 06:57:09.109259 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.099108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.22.113.62:37091)
0619 06:57:09.099959 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.099985 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.100037 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:57:09.101013 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.101038 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.101585 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.102011 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:09.102505 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.104059 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.104980 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.105690 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.105855 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.106920 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:09.106927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:09.106945 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:09.107535 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:09.107614 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:09.107642 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:09.107758 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.107915 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.108219 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":202,"threads_started":1}
I20260619 06:57:09.110834 8128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:09.111780 8128 log.cc:824] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:09.114012 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: No bootstrap required, opened a new log
I20260619 06:57:09.117862 8128 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } }
I20260619 06:57:09.118157 8128 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:09.118247 8128 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9fe7c141f8409ba6f444d6bdebbdf9, State: Initialized, Role: FOLLOWER
I20260619 06:57:09.118737 8128 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } }
I20260619 06:57:09.118898 8128 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:09.118988 8128 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:09.119117 8128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:09.120193 8128 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } }
I20260619 06:57:09.120582 8128 leader_election.cc:302] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff9fe7c141f8409ba6f444d6bdebbdf9; no voters:
I20260619 06:57:09.120975 8128 leader_election.cc:288] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:09.121204 8133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:09.121601 8133 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [term 1 LEADER]: Becoming Leader. State: Replica: ff9fe7c141f8409ba6f444d6bdebbdf9, State: Running, Role: LEADER
I20260619 06:57:09.122002 8133 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } }
I20260619 06:57:09.122535 8128 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:09.123951 8135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9fe7c141f8409ba6f444d6bdebbdf9. Latest consensus state: current_term: 1 leader_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } } }
I20260619 06:57:09.124019 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9fe7c141f8409ba6f444d6bdebbdf9" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 37091 } } }
I20260619 06:57:09.124102 8135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:09.124138 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:09.124575 8142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:09.128985 8142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:09.136225 8142 catalog_manager.cc:1347] Generated new cluster ID: 6e8dd820a39d4c09ae9e4e9ee00e4a8a
I20260619 06:57:09.136379 8142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:09.152568 8142 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:09.153726 8142 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:09.160930 8142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: Generated new TSK 0
I20260619 06:57:09.161774 8142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:09.170501 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--builtin_ntp_servers=127.22.113.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:09.295588 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:09.295902 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:09.295943 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:09.300262 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:09.300347 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:09.300412 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:09.305524 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:09.306950 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:09.308770 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:09.319932 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:09.319921 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.320168 8152 server_base.cc:1034] running on GCE node
W20260619 06:57:09.319900 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.320885 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:09.321640 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:09.322814 8152 hybrid_clock.cc:648] HybridClock initialized: now 1781852229322794 us; error 33 us; skew 500 ppm
I20260619 06:57:09.325140 8152 webserver.cc:460] Webserver started at http://127.22.113.1:40563/ using document root <none> and password file <none>
I20260619 06:57:09.325845 8152 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:09.325912 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:09.326156 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:09.328604 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "2fc04c3b52814b4e8f6c97b2ed29651d"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "7afca7c25b00648f04cce644ec0fe48d"
server_key_iv: "d74c182589af4345d2ba282c9dd77539"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.329252 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc04c3b52814b4e8f6c97b2ed29651d"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "7afca7c25b00648f04cce644ec0fe48d"
server_key_iv: "d74c182589af4345d2ba282c9dd77539"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.333709 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:09.336565 8166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.338124 8152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:09.338300 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "2fc04c3b52814b4e8f6c97b2ed29651d"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "7afca7c25b00648f04cce644ec0fe48d"
server_key_iv: "d74c182589af4345d2ba282c9dd77539"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.338435 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:09.351946 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:09.352774 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:09.352994 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:09.353677 8152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:09.354782 8152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:09.354853 8152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.354930 8152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:09.354974 8152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.366261 8152 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46709
I20260619 06:57:09.366362 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46709 every 8 connection(s)
I20260619 06:57:09.367393 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:09.368642 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8152
I20260619 06:57:09.368790 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:09.369182 22980 external_mini_cluster.cc:1428] Setting key 50d68de8712a4ea52ee6cc6ec625cea7
I20260619 06:57:09.371671 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--builtin_ntp_servers=127.22.113.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:09.379936 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.369872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:58255 (local address 127.22.113.62:37091)
0619 06:57:09.370060 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.370067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.371467 (+ 1400us) server_negotiation.cc:407] Connection header received
0619 06:57:09.373291 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.373294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.373345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.373438 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:09.374806 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.375536 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.377566 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.377774 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.377860 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.379408 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:09.379411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:09.379413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:09.379425 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:09.379446 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:09.379453 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:09.379525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.379756 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.379824 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:57:09.381465 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.370390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:37091 (local address 127.22.113.1:58255)
0619 06:57:09.371225 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.371333 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.372987 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.373657 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.373668 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.373794 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:09.374624 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.374647 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.375677 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.375684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.377426 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.377439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.377962 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.377969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.378230 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.378997 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:09.379000 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:09.379070 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:09.379171 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:09.379560 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:09.379572 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:09.379581 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.379748 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.380132 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":153,"threads_started":1}
I20260619 06:57:09.383021 8280 heartbeater.cc:347] Connected to a master server at 127.22.113.62:37091
I20260619 06:57:09.383369 8280 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:09.384013 8280 heartbeater.cc:511] Master 127.22.113.62:37091 requested a full tablet report, sending...
I20260619 06:57:09.385546 8092 ts_manager.cc:194] Registered new tserver with Master: 2fc04c3b52814b4e8f6c97b2ed29651d (127.22.113.1:46709)
I20260619 06:57:09.386723 8092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.1:58255
W20260619 06:57:09.503090 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:09.503489 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:09.503571 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:09.507831 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:09.507900 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:09.507990 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:09.512941 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:09.514256 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:09.516093 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:09.527343 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:09.527355 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.527663 8283 server_base.cc:1034] running on GCE node
W20260619 06:57:09.527717 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.528281 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:09.529117 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:09.530318 8283 hybrid_clock.cc:648] HybridClock initialized: now 1781852229530270 us; error 62 us; skew 500 ppm
I20260619 06:57:09.532501 8283 webserver.cc:460] Webserver started at http://127.22.113.2:36799/ using document root <none> and password file <none>
I20260619 06:57:09.533257 8283 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:09.533352 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:09.533620 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:09.536052 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "faa5a4690be748d58e23644ef67b1f6b"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "29af20725f6175ded73d412b89d01207"
server_key_iv: "6c68043ca895730e5b4507e1aa67a09c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.536684 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "faa5a4690be748d58e23644ef67b1f6b"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "29af20725f6175ded73d412b89d01207"
server_key_iv: "6c68043ca895730e5b4507e1aa67a09c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.541113 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:09.544171 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.545735 8283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:57:09.545882 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "faa5a4690be748d58e23644ef67b1f6b"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "29af20725f6175ded73d412b89d01207"
server_key_iv: "6c68043ca895730e5b4507e1aa67a09c"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.546008 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:09.563133 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:09.563947 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:09.564162 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:09.564837 8283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:09.565935 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:09.566020 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.566092 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:09.566144 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.577466 8283 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:41261
I20260619 06:57:09.577497 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:41261 every 8 connection(s)
I20260619 06:57:09.578588 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:09.579838 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8283
I20260619 06:57:09.579981 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:09.580353 22980 external_mini_cluster.cc:1428] Setting key 03850a58754b5ff4fd176b01a3fa382d
I20260619 06:57:09.582537 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--builtin_ntp_servers=127.22.113.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:09.590039 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.581038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:44841 (local address 127.22.113.62:37091)
0619 06:57:09.581199 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.581203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.582546 (+ 1343us) server_negotiation.cc:407] Connection header received
0619 06:57:09.584194 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.584199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.584264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.584375 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:09.585376 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.586263 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.587627 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.588068 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.588210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.589400 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:09.589405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:09.589410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:09.589430 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:09.589460 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:09.589474 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:09.589572 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.589738 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.589814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:57:09.590922 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.581508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:37091 (local address 127.22.113.2:44841)
0619 06:57:09.582322 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.582395 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.583961 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.584571 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.584579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.584666 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:09.585221 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.585232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.586422 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.586426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.587494 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.587504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.588230 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.588233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.588373 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.589109 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:09.589110 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:09.589155 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:09.589222 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:09.589577 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:09.589584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:09.589588 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.589696 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.589950 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":157,"threads_started":1}
I20260619 06:57:09.592384 8411 heartbeater.cc:347] Connected to a master server at 127.22.113.62:37091
I20260619 06:57:09.592774 8411 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:09.593631 8411 heartbeater.cc:511] Master 127.22.113.62:37091 requested a full tablet report, sending...
I20260619 06:57:09.595003 8092 ts_manager.cc:194] Registered new tserver with Master: faa5a4690be748d58e23644ef67b1f6b (127.22.113.2:41261)
I20260619 06:57:09.595749 8092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.2:44841
W20260619 06:57:09.702960 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:09.703219 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:09.703321 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:09.707239 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:09.707362 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:09.707449 8414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:09.712410 8414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:09.713582 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:09.714996 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:09.725328 8419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:09.725329 8420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:09.725363 8422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:09.725865 8414 server_base.cc:1034] running on GCE node
I20260619 06:57:09.726469 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:09.727195 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:09.728430 8414 hybrid_clock.cc:648] HybridClock initialized: now 1781852229728406 us; error 43 us; skew 500 ppm
I20260619 06:57:09.730659 8414 webserver.cc:460] Webserver started at http://127.22.113.3:42001/ using document root <none> and password file <none>
I20260619 06:57:09.731510 8414 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:09.731582 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:09.731812 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:09.734086 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "d677296de1cc4cbcbe1c0d782e2c3c73"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "f0c2cd423786caf4122d4efdd5c93012"
server_key_iv: "7fb5598530fa2b5a18726fc8f37c2fd5"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.734663 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "d677296de1cc4cbcbe1c0d782e2c3c73"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "f0c2cd423786caf4122d4efdd5c93012"
server_key_iv: "7fb5598530fa2b5a18726fc8f37c2fd5"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.738656 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:09.741621 8428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.743306 8414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:09.743455 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "d677296de1cc4cbcbe1c0d782e2c3c73"
format_stamp: "Formatted at 2026-06-19 06:57:09 on dist-test-slave-g8zg"
server_key: "f0c2cd423786caf4122d4efdd5c93012"
server_key_iv: "7fb5598530fa2b5a18726fc8f37c2fd5"
server_key_version: "encryptionkey@0"
I20260619 06:57:09.743578 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:09.769085 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:09.769816 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:09.770025 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:09.770646 8414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:09.771708 8414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:09.771780 8414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.771843 8414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:09.771888 8414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:09.781718 8414 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:37637
I20260619 06:57:09.781760 8541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:37637 every 8 connection(s)
I20260619 06:57:09.782608 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:09.790351 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8414
I20260619 06:57:09.790510 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:09.790892 22980 external_mini_cluster.cc:1428] Setting key dae8e7681dace0de380764d7ffe31a38
I20260619 06:57:09.793735 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.784872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:43149 (local address 127.22.113.62:37091)
0619 06:57:09.785069 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.785074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.786170 (+ 1096us) server_negotiation.cc:407] Connection header received
0619 06:57:09.787820 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.787823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.787877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.787970 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:09.789068 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.789609 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.791240 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.791617 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.791708 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.793192 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:09.793211 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:09.793217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:09.793232 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:09.793255 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:09.793266 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:09.793355 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.793572 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.793625 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:57:09.794782 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.785235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:37091 (local address 127.22.113.3:43149)
0619 06:57:09.785979 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.786030 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.787561 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.788224 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.788232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.788324 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:09.788914 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.788928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.789725 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.789727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.791104 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.791117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.791795 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.791799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.792013 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.792900 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:09.792902 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:09.792952 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:09.793026 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:09.793374 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:09.793389 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:09.793394 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.793555 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.793832 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260619 06:57:09.796103 8542 heartbeater.cc:347] Connected to a master server at 127.22.113.62:37091
I20260619 06:57:09.796348 8542 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:09.796911 8542 heartbeater.cc:511] Master 127.22.113.62:37091 requested a full tablet report, sending...
I20260619 06:57:09.798040 8092 ts_manager.cc:194] Registered new tserver with Master: d677296de1cc4cbcbe1c0d782e2c3c73 (127.22.113.3:37637)
I20260619 06:57:09.798650 8092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.3:43149
I20260619 06:57:09.806087 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:09.812783 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.808553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.22.113.62:37091)
0619 06:57:09.808743 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.808750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.808927 (+ 177us) server_negotiation.cc:407] Connection header received
0619 06:57:09.809107 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.809110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.809167 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.809251 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:09.809564 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.810128 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.811008 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.811319 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.811440 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.812245 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:09.812249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:09.812252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:09.812264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:09.812292 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:09.812301 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:09.812417 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.812545 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.812602 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:57:09.815996 8092 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:09.819943 8092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:09.836933 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.830046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46709 (local address 127.0.0.1:41300)
0619 06:57:09.830463 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.830480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.830583 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.832507 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.832510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.832530 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:09.832618 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.832623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.834016 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.834031 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.835409 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.835418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836463 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.836469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.836590 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.836620 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.836673 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.836770 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":111,"thread_start_us":107,"threads_started":1}
I20260619 06:57:09.836915 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.830948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.22.113.2:41261)
0619 06:57:09.831339 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.831354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.831380 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:57:09.831464 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.831472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.831685 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.832319 (+ 634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:09.832468 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.833921 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.835315 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836097 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.836215 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.836427 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.836623 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.836730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":184,"threads_started":1}
I20260619 06:57:09.837420 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.830952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.22.113.3:37637)
0619 06:57:09.831353 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.831365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.831388 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:57:09.831495 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.831500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.831757 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.832073 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:09.832399 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.833604 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.835436 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836155 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.836834 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.837061 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.837114 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.837220 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":155,"threads_started":1}
I20260619 06:57:09.837478 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.830577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41261 (local address 127.0.0.1:60958)
0619 06:57:09.830837 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.830853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.830947 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.831899 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.831907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.831933 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:09.832167 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.832176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.834038 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.834041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.835162 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.835171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836220 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.836224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.836345 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.836449 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.836573 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.837340 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":114,"thread_start_us":100,"threads_started":1}
I20260619 06:57:09.838055 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.829802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37637 (local address 127.0.0.1:44722)
0619 06:57:09.830252 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.830323 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.830474 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.831937 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.831941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.831967 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:09.832233 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.832241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.833729 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.833732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.835267 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.835278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836284 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.836287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.836376 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.836447 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.836560 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.836711 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":111,"thread_start_us":101,"threads_started":1}
I20260619 06:57:09.838725 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.831594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.22.113.1:46709)
0619 06:57:09.831986 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.831999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.832034 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:57:09.832113 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.832117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.832310 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.832505 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:09.832748 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.833866 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.835543 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.836312 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.838264 (+ 1952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.838455 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.838504 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.838605 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260619 06:57:09.841355 8214 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb5b36b3943460da5170871347b4410 (DEFAULT_TABLE table=test-table [id=7b7b4d6a9e5d4c8885c8a1c9ce092173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:09.841434 8476 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb5b36b3943460da5170871347b4410 (DEFAULT_TABLE table=test-table [id=7b7b4d6a9e5d4c8885c8a1c9ce092173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:09.841436 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb5b36b3943460da5170871347b4410 (DEFAULT_TABLE table=test-table [id=7b7b4d6a9e5d4c8885c8a1c9ce092173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:09.842599 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb5b36b3943460da5170871347b4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:09.842690 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb5b36b3943460da5170871347b4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:09.842690 8476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb5b36b3943460da5170871347b4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:09.849220 8559 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Bootstrap starting.
I20260619 06:57:09.850982 8561 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Bootstrap starting.
I20260619 06:57:09.851734 8559 tablet_bootstrap.cc:654] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:09.852689 8559 log.cc:824] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:09.853142 8561 tablet_bootstrap.cc:654] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:09.853545 8560 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Bootstrap starting.
I20260619 06:57:09.853938 8561 log.cc:824] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:09.855345 8559 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: No bootstrap required, opened a new log
I20260619 06:57:09.855604 8559 ts_tablet_manager.cc:1397] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:57:09.855671 8560 tablet_bootstrap.cc:654] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:09.856231 8561 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: No bootstrap required, opened a new log
I20260619 06:57:09.856436 8561 ts_tablet_manager.cc:1397] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:57:09.856421 8560 log.cc:824] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:09.858435 8560 tablet_bootstrap.cc:492] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: No bootstrap required, opened a new log
I20260619 06:57:09.858620 8560 ts_tablet_manager.cc:1397] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:57:09.859786 8559 raft_consensus.cc:348] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.860153 8559 raft_consensus.cc:374] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:09.860220 8559 raft_consensus.cc:729] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc04c3b52814b4e8f6c97b2ed29651d, State: Initialized, Role: FOLLOWER
I20260619 06:57:09.860729 8559 consensus_queue.cc:260] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.860946 8561 raft_consensus.cc:348] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.861238 8561 raft_consensus.cc:374] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:09.861310 8561 raft_consensus.cc:729] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d677296de1cc4cbcbe1c0d782e2c3c73, State: Initialized, Role: FOLLOWER
I20260619 06:57:09.861703 8280 heartbeater.cc:503] Master 127.22.113.62:37091 was elected leader, sending a full tablet report...
I20260619 06:57:09.861925 8559 ts_tablet_manager.cc:1428] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:57:09.862344 8561 consensus_queue.cc:260] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.862911 8560 raft_consensus.cc:348] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.863117 8542 heartbeater.cc:503] Master 127.22.113.62:37091 was elected leader, sending a full tablet report...
I20260619 06:57:09.863160 8560 raft_consensus.cc:374] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:09.863210 8560 raft_consensus.cc:729] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa5a4690be748d58e23644ef67b1f6b, State: Initialized, Role: FOLLOWER
I20260619 06:57:09.863621 8560 consensus_queue.cc:260] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.864617 8560 ts_tablet_manager.cc:1428] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:57:09.864920 8411 heartbeater.cc:503] Master 127.22.113.62:37091 was elected leader, sending a full tablet report...
I20260619 06:57:09.865320 8561 ts_tablet_manager.cc:1428] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260619 06:57:09.868788 8281 tablet.cc:2369] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:09.892120 8565 raft_consensus.cc:482] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:09.892480 8565 raft_consensus.cc:504] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.893958 8565 leader_election.cc:288] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d677296de1cc4cbcbe1c0d782e2c3c73 (127.22.113.3:37637), faa5a4690be748d58e23644ef67b1f6b (127.22.113.2:41261)
I20260619 06:57:09.898382 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.894373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:43211 (local address 127.22.113.2:41261)
0619 06:57:09.894531 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.894536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.894857 (+ 321us) server_negotiation.cc:407] Connection header received
0619 06:57:09.895009 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.895014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.895068 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.895154 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:09.895544 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.896053 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897330 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.897764 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897891 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.897949 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.898168 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.898240 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:57:09.898442 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.894269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:41261 (local address 127.22.113.1:43211)
0619 06:57:09.894728 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.894744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.894877 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.895184 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.895187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.895201 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:09.895416 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.895423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.896175 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.896177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.897200 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.897209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.897901 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.898010 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.898048 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.898126 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.898232 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":32,"thread_start_us":149,"threads_started":1}
I20260619 06:57:09.898427 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.894355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:52721 (local address 127.22.113.3:37637)
0619 06:57:09.894521 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.894527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.894733 (+ 206us) server_negotiation.cc:407] Connection header received
0619 06:57:09.894856 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.894860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.894926 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.895019 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:09.895392 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.895982 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897380 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.897784 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897934 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.898014 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.898229 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.898310 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:57:09.898478 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.894211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:37637 (local address 127.22.113.1:52721)
0619 06:57:09.894623 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.894636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:09.894740 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:09.895047 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:09.895050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:09.895068 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:09.895229 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.895236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.896109 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.896115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.897218 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:09.897228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.897972 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.897976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:09.898064 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.898086 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:57:09.898191 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:57:09.898275 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":109,"thread_start_us":115,"threads_started":1}
I20260619 06:57:09.899176 8496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb5b36b3943460da5170871347b4410" candidate_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" is_pre_election: true
I20260619 06:57:09.899580 8496 raft_consensus.cc:2454] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc04c3b52814b4e8f6c97b2ed29651d in term 0.
I20260619 06:57:09.899540 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb5b36b3943460da5170871347b4410" candidate_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa5a4690be748d58e23644ef67b1f6b" is_pre_election: true
I20260619 06:57:09.899902 8365 raft_consensus.cc:2454] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc04c3b52814b4e8f6c97b2ed29651d in term 0.
I20260619 06:57:09.900194 8169 leader_election.cc:302] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc04c3b52814b4e8f6c97b2ed29651d, d677296de1cc4cbcbe1c0d782e2c3c73; no voters:
I20260619 06:57:09.900650 8565 raft_consensus.cc:2781] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:09.900784 8565 raft_consensus.cc:482] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:09.900851 8565 raft_consensus.cc:3037] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:09.902036 8565 raft_consensus.cc:504] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.902446 8565 leader_election.cc:288] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [CANDIDATE]: Term 1 election: Requested vote from peers d677296de1cc4cbcbe1c0d782e2c3c73 (127.22.113.3:37637), faa5a4690be748d58e23644ef67b1f6b (127.22.113.2:41261)
I20260619 06:57:09.902896 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb5b36b3943460da5170871347b4410" candidate_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa5a4690be748d58e23644ef67b1f6b"
I20260619 06:57:09.902892 8496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb5b36b3943460da5170871347b4410" candidate_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73"
I20260619 06:57:09.903040 8365 raft_consensus.cc:3037] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:09.903007 8496 raft_consensus.cc:3037] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:09.904160 8365 raft_consensus.cc:2454] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc04c3b52814b4e8f6c97b2ed29651d in term 1.
I20260619 06:57:09.904460 8496 raft_consensus.cc:2454] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc04c3b52814b4e8f6c97b2ed29651d in term 1.
I20260619 06:57:09.904546 8168 leader_election.cc:302] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc04c3b52814b4e8f6c97b2ed29651d, faa5a4690be748d58e23644ef67b1f6b; no voters:
I20260619 06:57:09.904799 8565 raft_consensus.cc:2781] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:09.905108 8565 raft_consensus.cc:686] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 1 LEADER]: Becoming Leader. State: Replica: 2fc04c3b52814b4e8f6c97b2ed29651d, State: Running, Role: LEADER
I20260619 06:57:09.905560 8565 consensus_queue.cc:237] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } }
I20260619 06:57:09.909266 8092 catalog_manager.cc:5495] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc04c3b52814b4e8f6c97b2ed29651d (127.22.113.1). New cstate: current_term: 1 leader_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc04c3b52814b4e8f6c97b2ed29651d" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 } health_report { overall_health: UNKNOWN } } }
I20260619 06:57:09.937110 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.933508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.22.113.1:46709)
0619 06:57:09.933692 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:09.933697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:09.933713 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:09.933851 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:09.933854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:09.933912 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:09.934042 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:09.934282 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.934801 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.935677 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:09.935893 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:09.936002 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:09.936304 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:09.936716 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:09.936846 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:09.936908 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 06:57:09.936960 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:57:09.944689 8496 raft_consensus.cc:1261] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 1 FOLLOWER]: Refusing update from remote peer 2fc04c3b52814b4e8f6c97b2ed29651d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:09.944739 8365 raft_consensus.cc:1261] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 1 FOLLOWER]: Refusing update from remote peer 2fc04c3b52814b4e8f6c97b2ed29651d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:09.945524 8565 consensus_queue.cc:1035] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d677296de1cc4cbcbe1c0d782e2c3c73" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:09.945788 8570 consensus_queue.cc:1035] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa5a4690be748d58e23644ef67b1f6b" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:09.957444 8575 mvcc.cc:205] Tried to move back new op lower bound from 7298466733843464192 to 7298466733695844352. Current Snapshot: MvccSnapshot[applied={T|T < 7298466733843464192 or (T in {7298466733843464192})}]
I20260619 06:57:09.965893 8092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:09.966092 8092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:09.968652 8092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff9fe7c141f8409ba6f444d6bdebbdf9: Sending DeleteTablet for 3 replicas of tablet 5eb5b36b3943460da5170871347b4410
I20260619 06:57:09.969513 8214 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb5b36b3943460da5170871347b4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:09 UTC) from {username='slave'} at 127.0.0.1:41300
I20260619 06:57:09.969684 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb5b36b3943460da5170871347b4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:09 UTC) from {username='slave'} at 127.0.0.1:60958
I20260619 06:57:09.970096 8583 tablet_replica.cc:330] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d: stopping tablet replica
I20260619 06:57:09.970395 8584 tablet_replica.cc:330] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: stopping tablet replica
I20260619 06:57:09.970366 8476 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb5b36b3943460da5170871347b4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:09 UTC) from {username='slave'} at 127.0.0.1:44722
I20260619 06:57:09.970484 8583 raft_consensus.cc:2229] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:57:09.970784 8584 raft_consensus.cc:2229] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:09.970932 8583 raft_consensus.cc:2258] T 5eb5b36b3943460da5170871347b4410 P 2fc04c3b52814b4e8f6c97b2ed29651d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:09.971182 8584 raft_consensus.cc:2258] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:09.971735 8585 tablet_replica.cc:330] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: stopping tablet replica
I20260619 06:57:09.972097 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8152
I20260619 06:57:09.972103 8585 raft_consensus.cc:2229] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:09.973572 8585 raft_consensus.cc:2258] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:09.974726 8584 ts_tablet_manager.cc:1905] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:09.975358 8585 ts_tablet_manager.cc:1905] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:09.977241 8584 ts_tablet_manager.cc:1918] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:09.977358 8584 log.cc:1196] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/5eb5b36b3943460da5170871347b4410
I20260619 06:57:09.977663 8584 ts_tablet_manager.cc:1939] T 5eb5b36b3943460da5170871347b4410 P faa5a4690be748d58e23644ef67b1f6b: Deleting consensus metadata
I20260619 06:57:09.978437 8585 ts_tablet_manager.cc:1918] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:09.978544 8585 log.cc:1196] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/5eb5b36b3943460da5170871347b4410
I20260619 06:57:09.978856 8585 ts_tablet_manager.cc:1939] T 5eb5b36b3943460da5170871347b4410 P d677296de1cc4cbcbe1c0d782e2c3c73: Deleting consensus metadata
I20260619 06:57:09.979028 8077 catalog_manager.cc:4841] TS faa5a4690be748d58e23644ef67b1f6b (127.22.113.2:41261): tablet 5eb5b36b3943460da5170871347b4410 (table test-table [id=7b7b4d6a9e5d4c8885c8a1c9ce092173]) successfully deleted
I20260619 06:57:09.980273 8078 catalog_manager.cc:4841] TS d677296de1cc4cbcbe1c0d782e2c3c73 (127.22.113.3:37637): tablet 5eb5b36b3943460da5170871347b4410 (table test-table [id=7b7b4d6a9e5d4c8885c8a1c9ce092173]) successfully deleted
W20260619 06:57:09.981092 8077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:46709 (error 108)
I20260619 06:57:09.981272 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8283
I20260619 06:57:09.982686 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:09.982412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46709 (local address 127.0.0.1:41320)
0619 06:57:09.982541 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:09.982609 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:57:09.982863 8077 catalog_manager.cc:4584] TS 2fc04c3b52814b4e8f6c97b2ed29651d (127.22.113.1:46709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5eb5b36b3943460da5170871347b4410: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46709: connect: Connection refused (error 111)
I20260619 06:57:09.988749 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8414
I20260619 06:57:09.995501 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8061
2026-06-19T06:57:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:10 dist-test-slave-g8zg krb5kdc[8589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:10 dist-test-slave-g8zg krb5kdc[8589](info): set up 2 sockets
Jun 19 06:57:10 dist-test-slave-g8zg krb5kdc[8589](info): commencing operation
krb5kdc: starting...
W20260619 06:57:11.035514 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:11.176108 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40691
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:11.287026 8602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:11.287418 8602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:11.287508 8602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:11.292200 8602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:11.292285 8602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:11.292321 8602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:11.292358 8602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:11.292392 8602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:11.297940 8602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:40691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:11.299100 8602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:11.300190 8602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:11.309262 8610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:11.309510 8608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:11.309665 8602 server_base.cc:1034] running on GCE node
W20260619 06:57:11.309794 8607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:11.310382 8602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:11.311581 8602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:11.312757 8602 hybrid_clock.cc:648] HybridClock initialized: now 1781852231312733 us; error 41 us; skew 500 ppm
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:11.322105 8602 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:11.323163 8602 webserver.cc:460] Webserver started at http://127.22.113.62:45687/ using document root <none> and password file <none>
I20260619 06:57:11.323804 8602 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:11.323872 8602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:11.324043 8602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:11.326048 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "5816e0b1464f432786042500319c194d"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "cf20fd6e0569fcc25aee5daf8e86f3df"
server_key_iv: "f144121e26d42edb01584f32bf658596"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.326552 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "5816e0b1464f432786042500319c194d"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "cf20fd6e0569fcc25aee5daf8e86f3df"
server_key_iv: "f144121e26d42edb01584f32bf658596"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.330646 8602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:11.333446 8617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.334903 8602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:11.335062 8602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "5816e0b1464f432786042500319c194d"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "cf20fd6e0569fcc25aee5daf8e86f3df"
server_key_iv: "f144121e26d42edb01584f32bf658596"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.335191 8602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:11.354115 8602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:11.357395 8602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:11.357615 8602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:11.366158 8602 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:40691
I20260619 06:57:11.366338 8669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:40691 every 8 connection(s)
I20260619 06:57:11.367198 8602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:11.370708 8670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:11.374043 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8602
I20260619 06:57:11.374222 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:11.374630 22980 external_mini_cluster.cc:1428] Setting key e50ad7442f43d6e870c47785a4acd9f5
I20260619 06:57:11.378168 8670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: Bootstrap starting.
I20260619 06:57:11.380811 8670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:11.381669 8670 log.cc:824] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:11.384028 8670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: No bootstrap required, opened a new log
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:11.387871 8670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } }
I20260619 06:57:11.388111 8670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:11.388185 8670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5816e0b1464f432786042500319c194d, State: Initialized, Role: FOLLOWER
I20260619 06:57:11.388675 8670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } }
I20260619 06:57:11.388815 8670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:11.388906 8670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:11.389036 8670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:11.390165 8670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } }
I20260619 06:57:11.390555 8670 leader_election.cc:302] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5816e0b1464f432786042500319c194d; no voters:
I20260619 06:57:11.390892 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:11.376489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.22.113.62:40691)
0619 06:57:11.377055 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:11.377074 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:11.377115 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:57:11.377938 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:11.377959 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:11.378406 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:11.378768 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:11.379646 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.380674 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.381608 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.382152 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.382330 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:11.385266 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:11.385271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:11.385284 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:11.385312 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:11.387612 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.388060 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.388065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.388070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.388288 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.388566 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.388571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.388572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.389140 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:11.389345 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:11.389637 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:57:11.389938 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":180,"threads_started":1}
I20260619 06:57:11.390951 8670 leader_election.cc:288] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:11.391032 8675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:11.391296 8675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [term 1 LEADER]: Becoming Leader. State: Replica: 5816e0b1464f432786042500319c194d, State: Running, Role: LEADER
I20260619 06:57:11.391644 8675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } }
I20260619 06:57:11.392489 8670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:11.393584 8677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5816e0b1464f432786042500319c194d. Latest consensus state: current_term: 1 leader_uuid: "5816e0b1464f432786042500319c194d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } } }
I20260619 06:57:11.393764 8677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:11.393921 8676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5816e0b1464f432786042500319c194d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5816e0b1464f432786042500319c194d" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 40691 } } }
I20260619 06:57:11.394083 8676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:11.394511 8684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:11.398222 8684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:11.404930 8684 catalog_manager.cc:1347] Generated new cluster ID: ef155b0ec97843a3a4d94b7fa7abc55b
I20260619 06:57:11.405000 8684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:11.435556 8684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:11.436760 8684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:11.442422 8684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: Generated new TSK 0
I20260619 06:57:11.443362 8684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:57:11.483318 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:11.619897 8696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:11.620223 8696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:11.620273 8696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:11.624560 8696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:11.624634 8696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:11.624702 8696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:11.630374 8696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:11.631803 8696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:11.633002 8696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:11.644624 8702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:11.644641 8704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:11.644989 8696 server_base.cc:1034] running on GCE node
W20260619 06:57:11.644655 8701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:11.645638 8696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:11.646458 8696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:11.647682 8696 hybrid_clock.cc:648] HybridClock initialized: now 1781852231647631 us; error 65 us; skew 500 ppm
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:11.651098 8696 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:57:11.652266 8696 webserver.cc:460] Webserver started at http://127.22.113.1:42679/ using document root <none> and password file <none>
I20260619 06:57:11.652994 8696 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:11.653093 8696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:11.653381 8696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:11.655920 8696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "a28a146ccb7847638765c74f9aec3a45"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "182cdedf31da1acf8cbdf51fdaea79ac"
server_key_iv: "1e442624f460a53ab635b43d6c7978c0"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.656576 8696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "a28a146ccb7847638765c74f9aec3a45"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "182cdedf31da1acf8cbdf51fdaea79ac"
server_key_iv: "1e442624f460a53ab635b43d6c7978c0"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.661268 8696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:11.664654 8711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.666268 8696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:57:11.666468 8696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "a28a146ccb7847638765c74f9aec3a45"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "182cdedf31da1acf8cbdf51fdaea79ac"
server_key_iv: "1e442624f460a53ab635b43d6c7978c0"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.666610 8696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:11.681231 8696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:11.684855 8696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:11.685092 8696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:11.685834 8696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:11.686856 8696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:11.686931 8696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.687011 8696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:11.687054 8696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.699200 8696 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:36301
I20260619 06:57:11.699304 8824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:36301 every 8 connection(s)
I20260619 06:57:11.700433 8696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:11.702054 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8696
I20260619 06:57:11.702203 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:11.702534 22980 external_mini_cluster.cc:1428] Setting key 3206f4f51bf030e5a697df35f0c05386
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
I20260619 06:57:11.717804 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:11.702959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:37623 (local address 127.22.113.62:40691)
0619 06:57:11.703164 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:11.703169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:11.704418 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 06:57:11.705876 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:11.705879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:11.705929 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:11.706006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:11.707213 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.708050 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.709499 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.709901 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.710045 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:11.713367 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:11.713375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:11.713380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:11.713420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:11.714988 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.715715 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.715722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.715727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.715846 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.716286 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.716294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.716295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.716526 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:11.716642 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:11.717370 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 06:57:11.717558 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:57:11.718827 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:11.703599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40691 (local address 127.22.113.1:37623)
0619 06:57:11.704267 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:11.704318 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:11.705658 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:11.706184 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:11.706191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:11.706579 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:11.707075 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:11.707088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.708216 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.708221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:11.709337 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:11.709354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.710098 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.710101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:11.710337 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:11.711099 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:11.711118 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:11.713119 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:11.715130 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:11.715136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:11.715152 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:11.715572 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:11.715946 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:11.715948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:11.715950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:11.716113 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:11.716673 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:11.716683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:11.717065 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 06:57:11.717258 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 06:57:11.717644 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":167,"threads_started":1}
I20260619 06:57:11.720506 8825 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40691
I20260619 06:57:11.720850 8825 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:11.721633 8825 heartbeater.cc:511] Master 127.22.113.62:40691 requested a full tablet report, sending...
I20260619 06:57:11.723742 8632 ts_manager.cc:194] Registered new tserver with Master: a28a146ccb7847638765c74f9aec3a45 (127.22.113.1:36301)
I20260619 06:57:11.725478 8632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:37623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:57:11.735530 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:11.855834 8830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:11.856104 8830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:11.856150 8830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:11.860244 8830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:11.860312 8830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:11.860371 8830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:11.865708 8830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:11.866803 8830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:11.867923 8830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:11.878707 8836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:11.878695 8838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:11.878667 8835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:11.879209 8830 server_base.cc:1034] running on GCE node
I20260619 06:57:11.879725 8830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:11.880432 8830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:11.881637 8830 hybrid_clock.cc:648] HybridClock initialized: now 1781852231881612 us; error 44 us; skew 500 ppm
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:11.884845 8830 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:57:11.885814 8830 webserver.cc:460] Webserver started at http://127.22.113.2:33667/ using document root <none> and password file <none>
I20260619 06:57:11.886430 8830 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:11.886524 8830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:11.886770 8830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:11.888904 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "7d83ef58cf2e419592ef848d8fd9dacd"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "66d0f68420429df22909a6d3e54c5d5b"
server_key_iv: "14cd65e815df1febe6e8339d3c7d6563"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.889477 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "7d83ef58cf2e419592ef848d8fd9dacd"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "66d0f68420429df22909a6d3e54c5d5b"
server_key_iv: "14cd65e815df1febe6e8339d3c7d6563"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.893883 8830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 06:57:11.896670 8845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.898051 8830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:11.898200 8830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "7d83ef58cf2e419592ef848d8fd9dacd"
format_stamp: "Formatted at 2026-06-19 06:57:11 on dist-test-slave-g8zg"
server_key: "66d0f68420429df22909a6d3e54c5d5b"
server_key_iv: "14cd65e815df1febe6e8339d3c7d6563"
server_key_version: "encryptionkey@0"
I20260619 06:57:11.898315 8830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:11.924947 8830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:11.928009 8830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:11.928238 8830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:11.928853 8830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:11.929723 8830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:11.929800 8830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.929870 8830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:11.929916 8830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:11.940208 8830 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:42199
I20260619 06:57:11.940232 8958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:42199 every 8 connection(s)
I20260619 06:57:11.941128 8830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:11.943674 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8830
I20260619 06:57:11.943781 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:11.944096 22980 external_mini_cluster.cc:1428] Setting key 4cfadcae0a68b7d803238cf9cf667771
Jun 19 06:57:11 dist-test-slave-g8zg krb5kdc[8589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852231, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:57:11.957718 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:11.943126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:36703 (local address 127.22.113.62:40691)
0619 06:57:11.943342 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:11.943347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:11.944551 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 06:57:11.946136 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:11.946142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:11.946204 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:11.946306 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:11.947712 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.948298 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.949498 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.949741 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.949857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:11.953213 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:11.953216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:11.953219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:11.953244 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:11.954853 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.955610 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.955613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.955614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.955691 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:11.956205 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:11.956208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:11.956209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:11.956511 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:11.956629 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:11.957252 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 06:57:11.957424 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:57:11.958415 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:11.943717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40691 (local address 127.22.113.2:36703)
0619 06:57:11.944370 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:11.944431 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:11.945908 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:11.946487 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:11.946493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:11.946926 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:11.947557 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:11.947571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.948446 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.948450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:11.949370 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:11.949379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:11.949905 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:11.949909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:11.950082 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:11.950757 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:11.950774 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:11.952978 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:11.955014 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:11.955020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:11.955032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:11.955495 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:11.955814 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:11.955816 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:11.955818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:11.956031 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:11.956655 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:11.956661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:11.956965 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 06:57:11.957144 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:57:11.957450 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":156,"threads_started":1}
I20260619 06:57:11.959884 8959 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40691
I20260619 06:57:11.960140 8959 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:11.960638 8959 heartbeater.cc:511] Master 127.22.113.62:40691 requested a full tablet report, sending...
I20260619 06:57:11.961892 8632 ts_manager.cc:194] Registered new tserver with Master: 7d83ef58cf2e419592ef848d8fd9dacd (127.22.113.2:42199)
I20260619 06:57:11.962889 8632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:36703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:57:11.975010 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:12.100174 8964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:12.100446 8964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:12.100489 8964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:12.104579 8964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:12.104630 8964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:12.104686 8964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:12.109908 8964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:12.111045 8964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:12.112253 8964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:12.123427 8972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:12.123427 8970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:12.123453 8969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:12.123975 8964 server_base.cc:1034] running on GCE node
I20260619 06:57:12.124513 8964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:12.125264 8964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:12.126457 8964 hybrid_clock.cc:648] HybridClock initialized: now 1781852232126409 us; error 60 us; skew 500 ppm
Jun 19 06:57:12 dist-test-slave-g8zg krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852232, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:12.129863 8964 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:57:12.130976 8964 webserver.cc:460] Webserver started at http://127.22.113.3:41697/ using document root <none> and password file <none>
I20260619 06:57:12.131791 8964 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:12.131898 8964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:12.132185 8964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:12.134775 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "bb65fe877ff92ce607776e11eb5b862c"
server_key_iv: "9ffeaabc5c0a1aad4415c8995c96c78f"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.135547 8964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "bb65fe877ff92ce607776e11eb5b862c"
server_key_iv: "9ffeaabc5c0a1aad4415c8995c96c78f"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.140461 8964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:57:12.143529 8979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.145076 8964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:12.145277 8964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "bb65fe877ff92ce607776e11eb5b862c"
server_key_iv: "9ffeaabc5c0a1aad4415c8995c96c78f"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.145418 8964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:12.161430 8964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:12.165243 8964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:12.165494 8964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:12.166220 8964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:12.167197 8964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:12.167309 8964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.167383 8964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:12.167425 8964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.177549 8964 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:43319
I20260619 06:57:12.177618 9092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:43319 every 8 connection(s)
I20260619 06:57:12.178571 8964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:12.183130 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 8964
I20260619 06:57:12.183307 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:12.183598 22980 external_mini_cluster.cc:1428] Setting key 914fd4ad55d306cc2d5d443bc171ac06
Jun 19 06:57:12 dist-test-slave-g8zg krb5kdc[8589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852232, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:12.194394 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.181265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:40181 (local address 127.22.113.62:40691)
0619 06:57:12.181448 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.181452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.182306 (+ 854us) server_negotiation.cc:407] Connection header received
0619 06:57:12.183998 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.184002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.184067 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.184185 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:12.185525 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.186178 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.187390 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.187612 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.187728 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.190300 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:12.190309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:12.190311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:12.190334 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:12.191944 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:12.192535 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:12.192541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:12.192543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:12.192626 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:12.192977 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:12.192982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:12.192984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:12.193227 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:12.193330 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.194076 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.194189 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:57:12.195541 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.181609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:40691 (local address 127.22.113.3:40181)
0619 06:57:12.182097 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.182148 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.183752 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.184387 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.184395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.184807 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:12.185368 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.185381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.186307 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.186311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.187231 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.187267 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.187777 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.187782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.187962 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.188560 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:12.188576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:12.190126 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:12.192061 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:12.192067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:12.192086 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:12.192431 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:12.192726 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:12.192731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:12.192733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:12.192881 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:12.193435 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:12.193443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:12.193812 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.194030 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.194548 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260619 06:57:12.197216 9093 heartbeater.cc:347] Connected to a master server at 127.22.113.62:40691
I20260619 06:57:12.197575 9093 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:12.198256 9093 heartbeater.cc:511] Master 127.22.113.62:40691 requested a full tablet report, sending...
I20260619 06:57:12.199501 8632 ts_manager.cc:194] Registered new tserver with Master: ba532bdd843643e49ca3ef0dc0c7f6f4 (127.22.113.3:43319)
I20260619 06:57:12.200121 8632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:40181
I20260619 06:57:12.209973 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:12.221130 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.212463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.22.113.62:40691)
0619 06:57:12.212636 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.212641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.212898 (+ 257us) server_negotiation.cc:407] Connection header received
0619 06:57:12.213063 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.213066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.213127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.213227 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:12.214023 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.214608 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.215676 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.215904 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.216016 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.217661 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:12.217665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:12.217668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:12.217701 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:12.219132 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:12.219726 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:12.219729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:12.219731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:12.219814 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:12.220147 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:12.220150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:12.220152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:12.220386 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:12.220563 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.220835 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.220945 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:57:12.224609 8632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:12.228189 8632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:12.245853 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.238483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:42199 (local address 127.0.0.1:60874)
0619 06:57:12.238907 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.238957 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.239087 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.241027 (+ 1940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.241031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.241051 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:12.241198 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.241206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242863 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.242867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.244136 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.244147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.245265 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.245267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.245362 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.245398 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.245478 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.245666 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":89,"thread_start_us":113,"threads_started":1}
I20260619 06:57:12.245879 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.239318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.22.113.2:42199)
0619 06:57:12.239808 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.239826 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.239855 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:57:12.239946 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.239951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.240200 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.241056 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:12.241401 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242678 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.244288 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.245130 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.245275 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.245459 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.245526 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.245655 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":169,"threads_started":1}
I20260619 06:57:12.246646 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.239953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36301 (local address 127.0.0.1:59656)
0619 06:57:12.240206 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.240231 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.240327 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.241469 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.241471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.241484 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:12.241545 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.241549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242765 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.242768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.244513 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.244524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.246229 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.246232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.246319 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.246343 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.246428 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.246503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":231,"thread_start_us":101,"threads_started":1}
I20260619 06:57:12.246783 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.240567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.22.113.1:36301)
0619 06:57:12.240922 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.240929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.240950 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:57:12.241021 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.241026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.241183 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.241330 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:12.241706 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242630 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.244670 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.245590 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.245711 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.245872 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.246490 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.246611 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260619 06:57:12.247541 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.238523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:43319 (local address 127.0.0.1:44018)
0619 06:57:12.239362 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.239381 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.239489 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.240546 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.240552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.240576 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:12.240756 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.240764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242640 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.242645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.244513 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.244524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.247071 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.247075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.247182 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.247211 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.247300 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.247393 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":214,"thread_start_us":150,"threads_started":1}
I20260619 06:57:12.248272 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.238871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.22.113.3:43319)
0619 06:57:12.239922 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.239939 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.239965 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:57:12.240054 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.240062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.240349 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.240552 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:12.240922 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.242462 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.245950 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.246901 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.247904 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.248019 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.248062 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.248140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":177,"threads_started":1}
I20260619 06:57:12.249399 8893 tablet_service.cc:1467] Processing CreateTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 (DEFAULT_TABLE table=test-table [id=bb7e8314228b4bb58e3b1c0e26c7bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:12.250533 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:12.251484 9027 tablet_service.cc:1467] Processing CreateTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 (DEFAULT_TABLE table=test-table [id=bb7e8314228b4bb58e3b1c0e26c7bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:12.252599 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:12.256891 9110 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Bootstrap starting.
I20260619 06:57:12.257377 8759 tablet_service.cc:1467] Processing CreateTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 (DEFAULT_TABLE table=test-table [id=bb7e8314228b4bb58e3b1c0e26c7bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:12.258649 8759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:12.258996 9110 tablet_bootstrap.cc:654] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:12.260142 9110 log.cc:824] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:12.262148 9111 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Bootstrap starting.
I20260619 06:57:12.262329 9110 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: No bootstrap required, opened a new log
I20260619 06:57:12.262547 9110 ts_tablet_manager.cc:1397] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 06:57:12.264966 9111 tablet_bootstrap.cc:654] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:12.265460 9110 raft_consensus.cc:348] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.265719 9110 raft_consensus.cc:374] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:12.265772 9110 raft_consensus.cc:729] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d83ef58cf2e419592ef848d8fd9dacd, State: Initialized, Role: FOLLOWER
I20260619 06:57:12.266206 9110 consensus_queue.cc:260] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.266573 9111 log.cc:824] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:12.267009 8959 heartbeater.cc:503] Master 127.22.113.62:40691 was elected leader, sending a full tablet report...
I20260619 06:57:12.267189 9110 ts_tablet_manager.cc:1428] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 06:57:12.267388 9114 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: Bootstrap starting.
I20260619 06:57:12.269994 9114 tablet_bootstrap.cc:654] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:12.270447 9111 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: No bootstrap required, opened a new log
I20260619 06:57:12.270677 9111 ts_tablet_manager.cc:1397] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 06:57:12.271268 9114 log.cc:824] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:12.273420 9114 tablet_bootstrap.cc:492] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: No bootstrap required, opened a new log
I20260619 06:57:12.273660 9114 ts_tablet_manager.cc:1397] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 06:57:12.274163 9111 raft_consensus.cc:348] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.274389 9111 raft_consensus.cc:374] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:12.274422 9111 raft_consensus.cc:729] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba532bdd843643e49ca3ef0dc0c7f6f4, State: Initialized, Role: FOLLOWER
I20260619 06:57:12.274919 9111 consensus_queue.cc:260] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.275601 9093 heartbeater.cc:503] Master 127.22.113.62:40691 was elected leader, sending a full tablet report...
I20260619 06:57:12.275993 9111 ts_tablet_manager.cc:1428] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260619 06:57:12.278803 9114 raft_consensus.cc:348] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.279151 9114 raft_consensus.cc:374] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:12.279227 9114 raft_consensus.cc:729] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28a146ccb7847638765c74f9aec3a45, State: Initialized, Role: FOLLOWER
I20260619 06:57:12.279831 9114 consensus_queue.cc:260] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.280614 8825 heartbeater.cc:503] Master 127.22.113.62:40691 was elected leader, sending a full tablet report...
I20260619 06:57:12.280951 9114 ts_tablet_manager.cc:1428] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260619 06:57:12.341180 9118 raft_consensus.cc:482] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:12.341398 9118 raft_consensus.cc:504] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.342705 9118 leader_election.cc:288] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d83ef58cf2e419592ef848d8fd9dacd (127.22.113.2:42199), ba532bdd843643e49ca3ef0dc0c7f6f4 (127.22.113.3:43319)
I20260619 06:57:12.348915 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.343180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:60529 (local address 127.22.113.3:43319)
0619 06:57:12.343358 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.343363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.343826 (+ 463us) server_negotiation.cc:407] Connection header received
0619 06:57:12.343937 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.343941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.344010 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.344108 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:12.344560 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.345406 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.347395 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.348101 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.348251 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.348318 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.348657 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.348748 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 06:57:12.348915 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.343180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:44889 (local address 127.22.113.2:42199)
0619 06:57:12.343358 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.343363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.343762 (+ 399us) server_negotiation.cc:407] Connection header received
0619 06:57:12.343939 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.343942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.344013 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.344108 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:12.344596 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.345406 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.347395 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.348101 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.348251 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.348318 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.348657 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.348748 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 06:57:12.348948 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.343079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:42199 (local address 127.22.113.1:44889)
0619 06:57:12.343642 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.343659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.343800 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.344145 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.344148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.344176 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:12.344422 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.344430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.345605 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.345609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.347210 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.347222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.348260 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.348264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.348523 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.348540 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.348619 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.348730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":318,"spinlock_wait_cycles":15872,"thread_start_us":203,"threads_started":1}
I20260619 06:57:12.348946 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.343079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:43319 (local address 127.22.113.1:60529)
0619 06:57:12.343609 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.343627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.343800 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.344145 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.344148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.344176 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:12.344428 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.344434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.345605 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.345609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.347210 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.347222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.348260 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.348264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.348484 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.348529 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.348619 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.348730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":280,"spinlock_wait_cycles":31104,"thread_start_us":267,"threads_started":1}
I20260619 06:57:12.349982 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d75ec1b3e844cdba0f972ca7a1a2c9" candidate_uuid: "a28a146ccb7847638765c74f9aec3a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" is_pre_election: true
I20260619 06:57:12.350340 8913 raft_consensus.cc:2454] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a28a146ccb7847638765c74f9aec3a45 in term 0.
I20260619 06:57:12.350328 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d75ec1b3e844cdba0f972ca7a1a2c9" candidate_uuid: "a28a146ccb7847638765c74f9aec3a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" is_pre_election: true
I20260619 06:57:12.350668 9047 raft_consensus.cc:2454] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a28a146ccb7847638765c74f9aec3a45 in term 0.
I20260619 06:57:12.350919 8715 leader_election.cc:302] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d83ef58cf2e419592ef848d8fd9dacd, a28a146ccb7847638765c74f9aec3a45; no voters:
I20260619 06:57:12.351315 9118 raft_consensus.cc:2781] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:12.351410 9118 raft_consensus.cc:482] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:12.351444 9118 raft_consensus.cc:3037] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:12.352581 9118 raft_consensus.cc:504] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.352986 9118 leader_election.cc:288] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [CANDIDATE]: Term 1 election: Requested vote from peers 7d83ef58cf2e419592ef848d8fd9dacd (127.22.113.2:42199), ba532bdd843643e49ca3ef0dc0c7f6f4 (127.22.113.3:43319)
I20260619 06:57:12.353430 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d75ec1b3e844cdba0f972ca7a1a2c9" candidate_uuid: "a28a146ccb7847638765c74f9aec3a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4"
I20260619 06:57:12.353459 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d75ec1b3e844cdba0f972ca7a1a2c9" candidate_uuid: "a28a146ccb7847638765c74f9aec3a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d83ef58cf2e419592ef848d8fd9dacd"
I20260619 06:57:12.353565 9047 raft_consensus.cc:3037] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:12.353678 8913 raft_consensus.cc:3037] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:12.354915 9047 raft_consensus.cc:2454] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a28a146ccb7847638765c74f9aec3a45 in term 1.
I20260619 06:57:12.354915 8913 raft_consensus.cc:2454] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a28a146ccb7847638765c74f9aec3a45 in term 1.
I20260619 06:57:12.355324 8715 leader_election.cc:302] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d83ef58cf2e419592ef848d8fd9dacd, a28a146ccb7847638765c74f9aec3a45; no voters:
I20260619 06:57:12.355587 9118 raft_consensus.cc:2781] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:12.355826 9118 raft_consensus.cc:686] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [term 1 LEADER]: Becoming Leader. State: Replica: a28a146ccb7847638765c74f9aec3a45, State: Running, Role: LEADER
I20260619 06:57:12.356207 9118 consensus_queue.cc:237] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } }
I20260619 06:57:12.359480 8632 catalog_manager.cc:5495] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a28a146ccb7847638765c74f9aec3a45 (127.22.113.1). New cstate: current_term: 1 leader_uuid: "a28a146ccb7847638765c74f9aec3a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a28a146ccb7847638765c74f9aec3a45" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36301 } health_report { overall_health: HEALTHY } } }
I20260619 06:57:12.401382 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.397672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.22.113.1:36301)
0619 06:57:12.397852 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.397857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.397884 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:57:12.397978 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.397981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.398048 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.398165 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:12.398419 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.398934 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.399866 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.400088 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.400198 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.400522 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:12.400935 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:12.401102 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.401207 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.401247 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:57:12.408159 9047 raft_consensus.cc:1261] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 1 FOLLOWER]: Refusing update from remote peer a28a146ccb7847638765c74f9aec3a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:12.408181 8913 raft_consensus.cc:1261] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 1 FOLLOWER]: Refusing update from remote peer a28a146ccb7847638765c74f9aec3a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:12.408941 9118 consensus_queue.cc:1035] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba532bdd843643e49ca3ef0dc0c7f6f4" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:12.409152 9121 consensus_queue.cc:1035] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P a28a146ccb7847638765c74f9aec3a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d83ef58cf2e419592ef848d8fd9dacd" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:12.427559 8633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:12.427750 8633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:12.430593 8633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5816e0b1464f432786042500319c194d: Sending DeleteTablet for 3 replicas of tablet d0d75ec1b3e844cdba0f972ca7a1a2c9
I20260619 06:57:12.431524 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:12 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260619 06:57:12.431663 8759 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:12 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:59656
I20260619 06:57:12.431658 9027 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:12 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:44018
I20260619 06:57:12.432248 9136 tablet_replica.cc:330] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: stopping tablet replica
I20260619 06:57:12.432406 9134 tablet_replica.cc:330] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: stopping tablet replica
I20260619 06:57:12.432742 9136 raft_consensus.cc:2229] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:12.432765 9134 raft_consensus.cc:2229] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:12.433109 9134 raft_consensus.cc:2258] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:12.433139 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8696
I20260619 06:57:12.433266 9136 raft_consensus.cc:2258] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:12.436640 9136 ts_tablet_manager.cc:1905] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:12.437580 9134 ts_tablet_manager.cc:1905] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:12.440196 9134 ts_tablet_manager.cc:1918] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:12.440308 9134 log.cc:1196] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/d0d75ec1b3e844cdba0f972ca7a1a2c9
I20260619 06:57:12.440603 9134 ts_tablet_manager.cc:1939] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P 7d83ef58cf2e419592ef848d8fd9dacd: Deleting consensus metadata
I20260619 06:57:12.440762 9136 ts_tablet_manager.cc:1918] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:12.440876 9136 log.cc:1196] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/d0d75ec1b3e844cdba0f972ca7a1a2c9
I20260619 06:57:12.441239 9136 ts_tablet_manager.cc:1939] T d0d75ec1b3e844cdba0f972ca7a1a2c9 P ba532bdd843643e49ca3ef0dc0c7f6f4: Deleting consensus metadata
I20260619 06:57:12.441970 8621 catalog_manager.cc:4841] TS 7d83ef58cf2e419592ef848d8fd9dacd (127.22.113.2:42199): tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 (table test-table [id=bb7e8314228b4bb58e3b1c0e26c7bfa3]) successfully deleted
I20260619 06:57:12.442627 8618 catalog_manager.cc:4841] TS ba532bdd843643e49ca3ef0dc0c7f6f4 (127.22.113.3:43319): tablet d0d75ec1b3e844cdba0f972ca7a1a2c9 (table test-table [id=bb7e8314228b4bb58e3b1c0e26c7bfa3]) successfully deleted
W20260619 06:57:12.442878 8620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:36301 (error 108)
I20260619 06:57:12.443224 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8830
I20260619 06:57:12.447971 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.447596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36301 (local address 127.0.0.1:58680)
0619 06:57:12.447772 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.447863 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 06:57:12.448210 8620 catalog_manager.cc:4584] TS a28a146ccb7847638765c74f9aec3a45 (127.22.113.1:36301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0d75ec1b3e844cdba0f972ca7a1a2c9: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36301: connect: Connection refused (error 111)
I20260619 06:57:12.450259 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8964
I20260619 06:57:12.456432 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 8602
2026-06-19T06:57:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2461 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T06:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:12Z Disabled control of system clock
I20260619 06:57:12.491523 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:33745
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:33745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:12.616178 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:12.616461 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:12.616497 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:12.620571 9141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:12.620631 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:12.620647 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:12.620664 9141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:12.620680 9141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:12.625324 9141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:33745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:12.626478 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:12.628082 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:12.637447 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:12.637430 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:12.637447 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:12.637735 9141 server_base.cc:1034] running on GCE node
I20260619 06:57:12.638659 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:12.639997 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:12.641202 9141 hybrid_clock.cc:648] HybridClock initialized: now 1781852232641179 us; error 40 us; skew 500 ppm
I20260619 06:57:12.643393 9141 webserver.cc:460] Webserver started at http://127.22.113.62:38091/ using document root <none> and password file <none>
I20260619 06:57:12.644043 9141 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:12.644136 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:12.644385 9141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:12.646559 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "393541bd59342d0325ae022e376fcfed"
server_key_iv: "f7ebb809212ca83d7f8b9041b82e6bbf"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.647164 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "393541bd59342d0325ae022e376fcfed"
server_key_iv: "f7ebb809212ca83d7f8b9041b82e6bbf"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.651319 9141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:57:12.654218 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.655965 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:12.656139 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "393541bd59342d0325ae022e376fcfed"
server_key_iv: "f7ebb809212ca83d7f8b9041b82e6bbf"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.656275 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:12.667474 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:12.668191 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:12.668398 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:12.676537 9141 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:33745
I20260619 06:57:12.676559 9207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:33745 every 8 connection(s)
I20260619 06:57:12.677525 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:12.678849 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9141
I20260619 06:57:12.679000 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:12.679397 22980 external_mini_cluster.cc:1428] Setting key 131f6b97731e07290f8428041d45e5c7
I20260619 06:57:12.681895 9208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:12.688638 9208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: Bootstrap starting.
I20260619 06:57:12.689365 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.681138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.22.113.62:33745)
0619 06:57:12.681971 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.681995 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.682052 (+ 57us) server_negotiation.cc:407] Connection header received
0619 06:57:12.682735 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.682751 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.683186 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.683596 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:12.684031 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.685016 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.685878 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.686363 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.686534 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.687589 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:12.687594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:12.687605 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:12.688014 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:12.688072 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:12.688091 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:12.688201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.688329 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.688582 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":186,"threads_started":1}
I20260619 06:57:12.691370 9208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:12.692250 9208 log.cc:824] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:12.694482 9208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: No bootstrap required, opened a new log
I20260619 06:57:12.698483 9208 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } }
I20260619 06:57:12.698716 9208 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:12.698752 9208 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9559c2cc6a44f2da3c302e6a3d8cb73, State: Initialized, Role: FOLLOWER
I20260619 06:57:12.699194 9208 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } }
I20260619 06:57:12.699390 9208 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:12.699445 9208 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:12.699522 9208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:12.700518 9208 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } }
I20260619 06:57:12.700840 9208 leader_election.cc:302] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9559c2cc6a44f2da3c302e6a3d8cb73; no voters:
I20260619 06:57:12.701249 9208 leader_election.cc:288] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:12.701406 9213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:12.701735 9213 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [term 1 LEADER]: Becoming Leader. State: Replica: c9559c2cc6a44f2da3c302e6a3d8cb73, State: Running, Role: LEADER
I20260619 06:57:12.702143 9213 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } }
I20260619 06:57:12.702649 9208 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:12.704334 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } } }
I20260619 06:57:12.704368 9215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9559c2cc6a44f2da3c302e6a3d8cb73. Latest consensus state: current_term: 1 leader_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9559c2cc6a44f2da3c302e6a3d8cb73" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 33745 } } }
I20260619 06:57:12.704471 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:12.704492 9215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:12.704950 9224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:12.707947 9224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:12.713084 9224 catalog_manager.cc:1347] Generated new cluster ID: 77c5289988f443f59d5ec2ffe60dee13
I20260619 06:57:12.713152 9224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:12.732219 9224 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:12.733788 9224 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:12.739871 9224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: Generated new TSK 0
I20260619 06:57:12.740854 9224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:12.756592 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--builtin_ntp_servers=127.22.113.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:12.873376 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:12.873643 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:12.873682 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:12.877684 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:12.877739 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:12.877830 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:12.882592 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:12.883971 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:12.885424 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:12.895609 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:12.895560 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:12.895805 9232 server_base.cc:1034] running on GCE node
W20260619 06:57:12.896011 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:12.896433 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:12.897096 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:12.898298 9232 hybrid_clock.cc:648] HybridClock initialized: now 1781852232898275 us; error 43 us; skew 500 ppm
I20260619 06:57:12.900282 9232 webserver.cc:460] Webserver started at http://127.22.113.1:34337/ using document root <none> and password file <none>
I20260619 06:57:12.900869 9232 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:12.900962 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:12.901216 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:12.903453 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "c5eb0f09557641119f2b05a800dda42a"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "505b09a49aae9c8327592a9aa53b3d87"
server_key_iv: "8d70b71a22388d98c1081f19b2aa8d1e"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.904093 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "c5eb0f09557641119f2b05a800dda42a"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "505b09a49aae9c8327592a9aa53b3d87"
server_key_iv: "8d70b71a22388d98c1081f19b2aa8d1e"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.907963 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:57:12.910533 9246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.911900 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:12.912053 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "c5eb0f09557641119f2b05a800dda42a"
format_stamp: "Formatted at 2026-06-19 06:57:12 on dist-test-slave-g8zg"
server_key: "505b09a49aae9c8327592a9aa53b3d87"
server_key_iv: "8d70b71a22388d98c1081f19b2aa8d1e"
server_key_version: "encryptionkey@0"
I20260619 06:57:12.912166 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:12.939065 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:12.939811 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:12.940019 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:12.940627 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:12.941607 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:12.941680 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.941744 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:12.941797 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:12.951632 9232 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:34515
I20260619 06:57:12.951725 9359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:34515 every 8 connection(s)
I20260619 06:57:12.952591 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:12.953510 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9232
I20260619 06:57:12.953652 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:12.953974 22980 external_mini_cluster.cc:1428] Setting key 7a71238eb084b6a90d7300b08f1117ad
I20260619 06:57:12.956418 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--builtin_ntp_servers=127.22.113.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:12.965996 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.955001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:57933 (local address 127.22.113.62:33745)
0619 06:57:12.955156 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:12.955161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:12.956907 (+ 1746us) server_negotiation.cc:407] Connection header received
0619 06:57:12.959007 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:12.959010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:12.959062 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:12.959161 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:12.960467 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.961202 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.963397 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.963649 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.963748 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.965418 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:12.965421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:12.965423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:12.965437 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:12.965458 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:12.965465 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:12.965536 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:12.965795 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:57:12.965871 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:57:12.967571 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:12.955782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:33745 (local address 127.22.113.1:57933)
0619 06:57:12.956707 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:12.956767 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:12.958688 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:12.959434 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:12.959442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:12.959538 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:12.960299 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.960321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.961351 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.961357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.963220 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:12.963238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:12.963860 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:12.963866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:12.964147 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:12.965012 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:12.965015 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:12.965088 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:12.965203 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:12.965572 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:12.965584 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:12.965592 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:57:12.965779 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 06:57:12.966181 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":191,"threads_started":1}
I20260619 06:57:12.969146 9360 heartbeater.cc:347] Connected to a master server at 127.22.113.62:33745
I20260619 06:57:12.969480 9360 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:12.970309 9360 heartbeater.cc:511] Master 127.22.113.62:33745 requested a full tablet report, sending...
I20260619 06:57:12.972295 9172 ts_manager.cc:194] Registered new tserver with Master: c5eb0f09557641119f2b05a800dda42a (127.22.113.1:34515)
I20260619 06:57:12.973517 9172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.1:57933
W20260619 06:57:13.085448 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:13.085783 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:13.085863 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:13.090026 9363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:13.090108 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:13.090209 9363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:13.095170 9363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:13.096451 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:13.098032 9363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:13.108882 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:13.108875 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:13.108928 9371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:13.109148 9363 server_base.cc:1034] running on GCE node
I20260619 06:57:13.109686 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:13.110417 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:13.111626 9363 hybrid_clock.cc:648] HybridClock initialized: now 1781852233111609 us; error 39 us; skew 500 ppm
I20260619 06:57:13.113727 9363 webserver.cc:460] Webserver started at http://127.22.113.2:36157/ using document root <none> and password file <none>
I20260619 06:57:13.114380 9363 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:13.114473 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:13.114766 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:13.117159 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "2e0c8a8c3801482d846d6b100d4322fa"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "499fa0913b11434c55f466dbf5c1cd86"
server_key_iv: "48c5e3a6bcf82edcbfc18636a9f0d076"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.117717 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "2e0c8a8c3801482d846d6b100d4322fa"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "499fa0913b11434c55f466dbf5c1cd86"
server_key_iv: "48c5e3a6bcf82edcbfc18636a9f0d076"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.121527 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 06:57:13.124233 9377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.125707 9363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:13.125870 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "2e0c8a8c3801482d846d6b100d4322fa"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "499fa0913b11434c55f466dbf5c1cd86"
server_key_iv: "48c5e3a6bcf82edcbfc18636a9f0d076"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.126003 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:13.161257 9363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:13.161978 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:13.162175 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:13.162787 9363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:13.163836 9363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:13.163906 9363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.163980 9363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:13.164023 9363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.174099 9363 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:45873
I20260619 06:57:13.174168 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:45873 every 8 connection(s)
I20260619 06:57:13.175120 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:13.175742 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9363
I20260619 06:57:13.175892 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:13.176182 22980 external_mini_cluster.cc:1428] Setting key 63b58abb113b69667fde4cf1dfebe7ac
I20260619 06:57:13.178740 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--builtin_ntp_servers=127.22.113.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:13.186944 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.177637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:36895 (local address 127.22.113.62:33745)
0619 06:57:13.177820 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.177825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.179021 (+ 1196us) server_negotiation.cc:407] Connection header received
0619 06:57:13.180607 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.180610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.180664 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.180800 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:13.181980 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.182577 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.184504 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.184752 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.184856 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.186310 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:13.186313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:13.186315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:13.186331 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:13.186357 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:13.186366 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:13.186449 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.186705 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.186787 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:57:13.188112 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.178128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:33745 (local address 127.22.113.2:36895)
0619 06:57:13.178862 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.178914 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.180345 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.181013 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.181023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.181129 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:13.181804 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.181819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.182724 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.182731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.184331 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.184346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.184929 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.184937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.185172 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.185949 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:13.185953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:13.186010 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:13.186098 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:13.186479 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:13.186491 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:13.186500 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.186662 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.186974 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":148,"threads_started":1}
I20260619 06:57:13.189766 9491 heartbeater.cc:347] Connected to a master server at 127.22.113.62:33745
I20260619 06:57:13.190145 9491 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:13.190869 9491 heartbeater.cc:511] Master 127.22.113.62:33745 requested a full tablet report, sending...
I20260619 06:57:13.192250 9172 ts_manager.cc:194] Registered new tserver with Master: 2e0c8a8c3801482d846d6b100d4322fa (127.22.113.2:45873)
I20260619 06:57:13.193035 9172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.2:36895
W20260619 06:57:13.298785 9494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:13.299062 9494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:13.299100 9494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:13.303345 9494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:13.303414 9494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:13.303475 9494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:13.308454 9494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:13.309535 9494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:13.310873 9494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:13.321053 9500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:13.321122 9499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:13.321277 9494 server_base.cc:1034] running on GCE node
W20260619 06:57:13.321519 9502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:13.322073 9494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:13.322925 9494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:13.324126 9494 hybrid_clock.cc:648] HybridClock initialized: now 1781852233324086 us; error 56 us; skew 500 ppm
I20260619 06:57:13.326347 9494 webserver.cc:460] Webserver started at http://127.22.113.3:39693/ using document root <none> and password file <none>
I20260619 06:57:13.327016 9494 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:13.327082 9494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:13.327342 9494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:13.329810 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "3c4e3bceef134f3fad06c3aec3e71135"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "cd2d645a3b52acac606cd64f169b402c"
server_key_iv: "c4135c7e5677ee962dcf47f1d8f3d11e"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.330473 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "3c4e3bceef134f3fad06c3aec3e71135"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "cd2d645a3b52acac606cd64f169b402c"
server_key_iv: "c4135c7e5677ee962dcf47f1d8f3d11e"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.335435 9494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:57:13.338395 9508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.340119 9494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:13.340272 9494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "3c4e3bceef134f3fad06c3aec3e71135"
format_stamp: "Formatted at 2026-06-19 06:57:13 on dist-test-slave-g8zg"
server_key: "cd2d645a3b52acac606cd64f169b402c"
server_key_iv: "c4135c7e5677ee962dcf47f1d8f3d11e"
server_key_version: "encryptionkey@0"
I20260619 06:57:13.340425 9494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:13.355396 9494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:13.356207 9494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:13.356434 9494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:13.357182 9494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:13.358300 9494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:13.358387 9494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.358465 9494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:13.358507 9494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:13.369539 9494 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:43549
I20260619 06:57:13.369596 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:43549 every 8 connection(s)
I20260619 06:57:13.370659 9494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:13.377493 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9494
I20260619 06:57:13.377662 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:13.378050 22980 external_mini_cluster.cc:1428] Setting key e7074e70117886864a46fc653cb16a06
I20260619 06:57:13.381294 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.372895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:57433 (local address 127.22.113.62:33745)
0619 06:57:13.373082 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.373087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.374313 (+ 1226us) server_negotiation.cc:407] Connection header received
0619 06:57:13.375900 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.375903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.375964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.376077 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:13.376985 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.377540 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.378963 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.379373 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.379488 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.380781 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:13.380805 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:13.380809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:13.380830 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:13.380851 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:13.380860 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:13.380927 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.381100 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.381162 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:57:13.382131 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.373410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:33745 (local address 127.22.113.3:57433)
0619 06:57:13.374142 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.374187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.375683 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.376256 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.376263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.376342 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:13.376838 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.376850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.377676 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.377679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.378816 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.378826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.379558 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.379564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.379754 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.380465 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:13.380467 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:13.380538 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:13.380619 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:13.380959 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:13.380967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:13.380971 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.381079 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.381290 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260619 06:57:13.383460 9622 heartbeater.cc:347] Connected to a master server at 127.22.113.62:33745
I20260619 06:57:13.383713 9622 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:13.384178 9622 heartbeater.cc:511] Master 127.22.113.62:33745 requested a full tablet report, sending...
I20260619 06:57:13.385238 9172 ts_manager.cc:194] Registered new tserver with Master: 3c4e3bceef134f3fad06c3aec3e71135 (127.22.113.3:43549)
I20260619 06:57:13.385780 9172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.3:57433
I20260619 06:57:13.392771 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:13.399004 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.394851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.22.113.62:33745)
0619 06:57:13.395055 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.395060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.395350 (+ 290us) server_negotiation.cc:407] Connection header received
0619 06:57:13.395465 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.395468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.395519 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.395593 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:13.395899 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.396528 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.397357 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.397579 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.397672 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.398451 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:13.398454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:13.398457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:13.398468 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:13.398488 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:13.398495 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:13.398593 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.398730 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.398801 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:57:13.402397 9172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:13.406476 9172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:13.423772 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.417094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.22.113.3:43549)
0619 06:57:13.417761 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.417792 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.417823 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:57:13.417913 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.417919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.418157 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.418324 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:13.418903 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420233 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.422266 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.422912 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423037 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.423224 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.423459 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.423568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":156,"threads_started":1}
I20260619 06:57:13.423763 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.417353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34515 (local address 127.0.0.1:49518)
0619 06:57:13.418129 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.418145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.418238 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.419026 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.419028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.419046 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:13.419152 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.419159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420692 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.420696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.422092 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.422104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.423186 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.423282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.423309 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.423379 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.423565 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":123,"thread_start_us":145,"threads_started":1}
I20260619 06:57:13.424225 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.418016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.22.113.1:34515)
0619 06:57:13.418460 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.418491 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.418520 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:57:13.418610 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.418615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.418858 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.419408 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:13.419547 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420574 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.422198 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.423058 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423658 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.423866 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.423924 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.424045 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":114,"threads_started":1}
I20260619 06:57:13.424242 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.416938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:43549 (local address 127.0.0.1:46168)
0619 06:57:13.417395 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.417456 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.417615 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.418514 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.418521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.418548 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:13.418745 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.418754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420374 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.420378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.422104 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.422115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.423056 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.423199 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.423282 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.423406 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.423564 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":341,"thread_start_us":108,"threads_started":1}
I20260619 06:57:13.424549 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.418346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45873 (local address 127.0.0.1:56534)
0619 06:57:13.418613 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.418626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.418712 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.419625 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.419628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.419645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:13.419745 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.419760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420904 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.420909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.422672 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.422681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.423935 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.424042 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.424067 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.424135 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.424222 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":93,"thread_start_us":108,"threads_started":1}
I20260619 06:57:13.424558 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.418021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.22.113.2:45873)
0619 06:57:13.419127 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.419138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.419160 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:57:13.419296 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.419304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.419473 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.419630 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:13.419859 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.420807 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.422804 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.423587 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.423695 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.423859 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.424302 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.424392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":135,"threads_started":1}
I20260619 06:57:13.427337 9556 tablet_service.cc:1467] Processing CreateTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a (DEFAULT_TABLE table=test-table [id=bea066337d4c40fb84848673af9e7b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:13.427493 9294 tablet_service.cc:1467] Processing CreateTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a (DEFAULT_TABLE table=test-table [id=bea066337d4c40fb84848673af9e7b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:13.427896 9425 tablet_service.cc:1467] Processing CreateTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a (DEFAULT_TABLE table=test-table [id=bea066337d4c40fb84848673af9e7b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:13.428478 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:13.428478 9294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:13.428937 9425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:13.436316 9639 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Bootstrap starting.
I20260619 06:57:13.436848 9640 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Bootstrap starting.
I20260619 06:57:13.438024 9641 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Bootstrap starting.
I20260619 06:57:13.439749 9640 tablet_bootstrap.cc:654] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:13.440199 9641 tablet_bootstrap.cc:654] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:13.440860 9640 log.cc:824] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:13.441226 9641 log.cc:824] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:13.441526 9639 tablet_bootstrap.cc:654] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:13.442356 9639 log.cc:824] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:13.443399 9640 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: No bootstrap required, opened a new log
I20260619 06:57:13.443629 9640 ts_tablet_manager.cc:1397] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:57:13.443917 9641 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: No bootstrap required, opened a new log
I20260619 06:57:13.444147 9641 ts_tablet_manager.cc:1397] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260619 06:57:13.444615 9639 tablet_bootstrap.cc:492] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: No bootstrap required, opened a new log
I20260619 06:57:13.444825 9639 ts_tablet_manager.cc:1397] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 06:57:13.447213 9640 raft_consensus.cc:348] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.447578 9640 raft_consensus.cc:374] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:13.447662 9640 raft_consensus.cc:729] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5eb0f09557641119f2b05a800dda42a, State: Initialized, Role: FOLLOWER
I20260619 06:57:13.448151 9640 consensus_queue.cc:260] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.448990 9360 heartbeater.cc:503] Master 127.22.113.62:33745 was elected leader, sending a full tablet report...
I20260619 06:57:13.448928 9641 raft_consensus.cc:348] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.449226 9641 raft_consensus.cc:374] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:13.449298 9641 raft_consensus.cc:729] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0c8a8c3801482d846d6b100d4322fa, State: Initialized, Role: FOLLOWER
I20260619 06:57:13.449333 9640 ts_tablet_manager.cc:1428] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:57:13.449759 9641 consensus_queue.cc:260] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.450459 9491 heartbeater.cc:503] Master 127.22.113.62:33745 was elected leader, sending a full tablet report...
I20260619 06:57:13.450789 9641 ts_tablet_manager.cc:1428] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260619 06:57:13.451097 9639 raft_consensus.cc:348] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.451416 9639 raft_consensus.cc:374] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:13.451493 9639 raft_consensus.cc:729] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4e3bceef134f3fad06c3aec3e71135, State: Initialized, Role: FOLLOWER
I20260619 06:57:13.451912 9639 consensus_queue.cc:260] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.452560 9622 heartbeater.cc:503] Master 127.22.113.62:33745 was elected leader, sending a full tablet report...
I20260619 06:57:13.452843 9639 ts_tablet_manager.cc:1428] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260619 06:57:13.453768 9361 tablet.cc:2369] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:13.460167 9647 raft_consensus.cc:482] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:13.460427 9647 raft_consensus.cc:504] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.461731 9647 leader_election.cc:288] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e0c8a8c3801482d846d6b100d4322fa (127.22.113.2:45873), c5eb0f09557641119f2b05a800dda42a (127.22.113.1:34515)
I20260619 06:57:13.466853 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.462512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:32893 (local address 127.22.113.1:34515)
0619 06:57:13.462641 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.462646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.462660 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:57:13.462718 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.462721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.462762 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.462826 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:13.463094 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.463889 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.465483 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.466105 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.466240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.466291 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.466603 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.466693 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:57:13.466858 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.461956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:45873 (local address 127.22.113.3:33895)
0619 06:57:13.462156 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.462174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.462269 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.462610 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.462614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.462629 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:13.462766 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.462772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.463891 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.463893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.465345 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.465352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.466237 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.466240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.466414 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.466490 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.466580 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.466730 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":220}
I20260619 06:57:13.466886 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.462086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34515 (local address 127.22.113.3:32893)
0619 06:57:13.462469 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.462483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:13.462603 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:13.462868 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:13.462871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:13.462891 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:13.462989 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.462993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.463993 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.463996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.465342 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:13.465352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.466237 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.466240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:13.466438 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.466487 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 06:57:13.466571 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:57:13.466682 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":118,"thread_start_us":152,"threads_started":1}
I20260619 06:57:13.467438 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.462083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:33895 (local address 127.22.113.2:45873)
0619 06:57:13.462250 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.462255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.462271 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:13.462407 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.462410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.462467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.462545 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:13.462909 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.463781 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.465483 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.466105 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.466235 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.466291 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.467212 (+ 921us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.467320 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:57:13.467768 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8901a963e5b4ab3bc2fd0380b6f4d7a" candidate_uuid: "3c4e3bceef134f3fad06c3aec3e71135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5eb0f09557641119f2b05a800dda42a" is_pre_election: true
I20260619 06:57:13.468129 9314 raft_consensus.cc:2454] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e3bceef134f3fad06c3aec3e71135 in term 0.
I20260619 06:57:13.468129 9445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8901a963e5b4ab3bc2fd0380b6f4d7a" candidate_uuid: "3c4e3bceef134f3fad06c3aec3e71135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0c8a8c3801482d846d6b100d4322fa" is_pre_election: true
I20260619 06:57:13.468375 9445 raft_consensus.cc:2454] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c4e3bceef134f3fad06c3aec3e71135 in term 0.
I20260619 06:57:13.468644 9511 leader_election.cc:302] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4e3bceef134f3fad06c3aec3e71135, c5eb0f09557641119f2b05a800dda42a; no voters:
I20260619 06:57:13.468937 9647 raft_consensus.cc:2781] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:13.469022 9647 raft_consensus.cc:482] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:13.469053 9647 raft_consensus.cc:3037] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:13.470268 9647 raft_consensus.cc:504] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.470676 9647 leader_election.cc:288] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [CANDIDATE]: Term 1 election: Requested vote from peers 2e0c8a8c3801482d846d6b100d4322fa (127.22.113.2:45873), c5eb0f09557641119f2b05a800dda42a (127.22.113.1:34515)
I20260619 06:57:13.471062 9445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8901a963e5b4ab3bc2fd0380b6f4d7a" candidate_uuid: "3c4e3bceef134f3fad06c3aec3e71135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0c8a8c3801482d846d6b100d4322fa"
I20260619 06:57:13.471117 9314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8901a963e5b4ab3bc2fd0380b6f4d7a" candidate_uuid: "3c4e3bceef134f3fad06c3aec3e71135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5eb0f09557641119f2b05a800dda42a"
I20260619 06:57:13.471210 9445 raft_consensus.cc:3037] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:13.471210 9314 raft_consensus.cc:3037] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:13.472306 9314 raft_consensus.cc:2454] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e3bceef134f3fad06c3aec3e71135 in term 1.
I20260619 06:57:13.472344 9445 raft_consensus.cc:2454] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c4e3bceef134f3fad06c3aec3e71135 in term 1.
I20260619 06:57:13.472661 9511 leader_election.cc:302] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4e3bceef134f3fad06c3aec3e71135, c5eb0f09557641119f2b05a800dda42a; no voters:
I20260619 06:57:13.472892 9647 raft_consensus.cc:2781] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:13.473105 9647 raft_consensus.cc:686] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4e3bceef134f3fad06c3aec3e71135, State: Running, Role: LEADER
I20260619 06:57:13.473445 9647 consensus_queue.cc:237] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } }
I20260619 06:57:13.476538 9170 catalog_manager.cc:5495] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c4e3bceef134f3fad06c3aec3e71135 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "3c4e3bceef134f3fad06c3aec3e71135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c4e3bceef134f3fad06c3aec3e71135" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 } health_report { overall_health: UNKNOWN } } }
I20260619 06:57:13.487984 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.484621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.22.113.3:43549)
0619 06:57:13.484745 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:13.484749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:13.484762 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:57:13.484844 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:13.484846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:13.484892 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:13.484981 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:13.485207 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.485682 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.486545 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:13.486747 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:13.486822 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:13.487095 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:13.487533 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:13.487651 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:13.487748 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:57:13.487820 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:57:13.494510 9445 raft_consensus.cc:1261] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 1 FOLLOWER]: Refusing update from remote peer 3c4e3bceef134f3fad06c3aec3e71135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:13.494539 9314 raft_consensus.cc:1261] T f8901a963e5b4ab3bc2fd0380b6f4d7a P c5eb0f09557641119f2b05a800dda42a [term 1 FOLLOWER]: Refusing update from remote peer 3c4e3bceef134f3fad06c3aec3e71135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:13.495366 9649 consensus_queue.cc:1035] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e0c8a8c3801482d846d6b100d4322fa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:13.495602 9647 consensus_queue.cc:1035] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5eb0f09557641119f2b05a800dda42a" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:13.504411 9652 mvcc.cc:205] Tried to move back new op lower bound from 7298466748385284096 to 7298466748310237184. Current Snapshot: MvccSnapshot[applied={T|T < 7298466748385284096}]
I20260619 06:57:13.504442 9653 mvcc.cc:205] Tried to move back new op lower bound from 7298466748385284096 to 7298466748310237184. Current Snapshot: MvccSnapshot[applied={T|T < 7298466748385284096}]
I20260619 06:57:13.504603 9654 mvcc.cc:205] Tried to move back new op lower bound from 7298466748385284096 to 7298466748310237184. Current Snapshot: MvccSnapshot[applied={T|T < 7298466748385284096}]
I20260619 06:57:13.519160 9172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:13.519421 9172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:13.522919 9172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9559c2cc6a44f2da3c302e6a3d8cb73: Sending DeleteTablet for 3 replicas of tablet f8901a963e5b4ab3bc2fd0380b6f4d7a
I20260619 06:57:13.524346 9556 tablet_service.cc:1514] Processing DeleteTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:13 UTC) from {username='slave'} at 127.0.0.1:46168
I20260619 06:57:13.524354 9425 tablet_service.cc:1514] Processing DeleteTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20260619 06:57:13.524961 9664 tablet_replica.cc:330] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: stopping tablet replica
I20260619 06:57:13.525189 9294 tablet_service.cc:1514] Processing DeleteTablet for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:13 UTC) from {username='slave'} at 127.0.0.1:49518
I20260619 06:57:13.525375 9664 raft_consensus.cc:2229] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:13.525775 9664 raft_consensus.cc:2258] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:13.526196 9663 tablet_replica.cc:330] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: stopping tablet replica
I20260619 06:57:13.526361 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9232
I20260619 06:57:13.526521 9663 raft_consensus.cc:2229] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:57:13.526960 9663 raft_consensus.cc:2258] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:13.529300 9664 ts_tablet_manager.cc:1905] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:13.530997 9663 ts_tablet_manager.cc:1905] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:13.532924 9664 ts_tablet_manager.cc:1918] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:13.533098 9664 log.cc:1196] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/f8901a963e5b4ab3bc2fd0380b6f4d7a
I20260619 06:57:13.533502 9664 ts_tablet_manager.cc:1939] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 2e0c8a8c3801482d846d6b100d4322fa: Deleting consensus metadata
I20260619 06:57:13.534986 9663 ts_tablet_manager.cc:1918] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:13.534988 9159 catalog_manager.cc:4841] TS 2e0c8a8c3801482d846d6b100d4322fa (127.22.113.2:45873): tablet f8901a963e5b4ab3bc2fd0380b6f4d7a (table test-table [id=bea066337d4c40fb84848673af9e7b18]) successfully deleted
I20260619 06:57:13.535094 9663 log.cc:1196] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/f8901a963e5b4ab3bc2fd0380b6f4d7a
I20260619 06:57:13.535468 9663 ts_tablet_manager.cc:1939] T f8901a963e5b4ab3bc2fd0380b6f4d7a P 3c4e3bceef134f3fad06c3aec3e71135: Deleting consensus metadata
W20260619 06:57:13.536609 9158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:34515 (error 108)
I20260619 06:57:13.536851 9159 catalog_manager.cc:4841] TS 3c4e3bceef134f3fad06c3aec3e71135 (127.22.113.3:43549): tablet f8901a963e5b4ab3bc2fd0380b6f4d7a (table test-table [id=bea066337d4c40fb84848673af9e7b18]) successfully deleted
I20260619 06:57:13.537613 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9363
I20260619 06:57:13.537901 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:13.537541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:34515 (local address 127.0.0.1:49526)
0619 06:57:13.537692 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:13.537775 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:57:13.541826 9158 catalog_manager.cc:4584] TS c5eb0f09557641119f2b05a800dda42a (127.22.113.1:34515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8901a963e5b4ab3bc2fd0380b6f4d7a: Network error: Client connection negotiation failed: client connection to 127.22.113.1:34515: connect: Connection refused (error 111)
I20260619 06:57:13.547397 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9494
I20260619 06:57:13.553560 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9141
2026-06-19T06:57:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:13 dist-test-slave-g8zg krb5kdc[9669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:13 dist-test-slave-g8zg krb5kdc[9669](info): set up 2 sockets
Jun 19 06:57:13 dist-test-slave-g8zg krb5kdc[9669](info): commencing operation
krb5kdc: starting...
W20260619 06:57:14.603628 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:14 dist-test-slave-g8zg krb5kdc[9669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:14.739568 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42555
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:14.851707 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:14.851972 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:14.852015 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:14.855979 9682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:14.856041 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:14.856061 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:14.856077 9682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:14.856093 9682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:14.861179 9682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:42555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:14.862159 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:14.863106 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:14.872009 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:14.871994 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:14.872229 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:14.872520 9682 server_base.cc:1034] running on GCE node
I20260619 06:57:14.873135 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:14.874251 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:14.875422 9682 hybrid_clock.cc:648] HybridClock initialized: now 1781852234875381 us; error 57 us; skew 500 ppm
Jun 19 06:57:14 dist-test-slave-g8zg krb5kdc[9669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852234, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:14.878249 9682 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:14.879221 9682 webserver.cc:460] Webserver started at http://127.22.113.62:36727/ using document root <none> and password file <none>
I20260619 06:57:14.879840 9682 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:14.879904 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:14.880082 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:14.882117 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "ab3e6d1949f640549ca92550efeb00c1"
format_stamp: "Formatted at 2026-06-19 06:57:14 on dist-test-slave-g8zg"
server_key: "d0d8d5ec81fcfedc98371fd497e2017b"
server_key_iv: "4a1f2e6efc5cad0536363e0b6a1a487e"
server_key_version: "encryptionkey@0"
I20260619 06:57:14.882627 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "ab3e6d1949f640549ca92550efeb00c1"
format_stamp: "Formatted at 2026-06-19 06:57:14 on dist-test-slave-g8zg"
server_key: "d0d8d5ec81fcfedc98371fd497e2017b"
server_key_iv: "4a1f2e6efc5cad0536363e0b6a1a487e"
server_key_version: "encryptionkey@0"
I20260619 06:57:14.886559 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:14.889230 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:14.890698 9682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:57:14.890862 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "ab3e6d1949f640549ca92550efeb00c1"
format_stamp: "Formatted at 2026-06-19 06:57:14 on dist-test-slave-g8zg"
server_key: "d0d8d5ec81fcfedc98371fd497e2017b"
server_key_iv: "4a1f2e6efc5cad0536363e0b6a1a487e"
server_key_version: "encryptionkey@0"
I20260619 06:57:14.890985 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:14.900787 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:14.903666 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:14.903872 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:14.911495 9682 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:42555
I20260619 06:57:14.911530 9749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:42555 every 8 connection(s)
I20260619 06:57:14.912456 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:14.915804 9750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:14.916841 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9682
I20260619 06:57:14.916997 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:14.917351 22980 external_mini_cluster.cc:1428] Setting key faf2ffc6abd6d4f6b21d35febdc82b51
I20260619 06:57:14.922839 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: Bootstrap starting.
I20260619 06:57:14.925992 9750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:14.927043 9750 log.cc:824] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:14 dist-test-slave-g8zg krb5kdc[9669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:14.929788 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: No bootstrap required, opened a new log
I20260619 06:57:14.933876 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:14.919045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.22.113.62:42555)
0619 06:57:14.919615 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:14.919632 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:14.919672 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:57:14.920476 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:14.920496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:14.920865 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:14.921242 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:14.922074 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:14.923186 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:14.924618 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:14.925112 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:14.925238 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:14.928496 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:14.928502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:14.928520 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:14.928551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:14.930542 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:14.930972 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:14.930977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:14.930983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:14.931128 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:14.931475 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:14.931478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:14.931480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:14.932014 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:14.932234 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:14.932494 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 06:57:14.932841 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":191,"threads_started":1}
I20260619 06:57:14.934686 9750 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } }
I20260619 06:57:14.934945 9750 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:14.935006 9750 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3e6d1949f640549ca92550efeb00c1, State: Initialized, Role: FOLLOWER
I20260619 06:57:14.935530 9750 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } }
I20260619 06:57:14.935688 9750 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:14.935762 9750 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:14.935866 9750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:14.937055 9750 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } }
I20260619 06:57:14.937400 9750 leader_election.cc:302] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3e6d1949f640549ca92550efeb00c1; no voters:
I20260619 06:57:14.937717 9750 leader_election.cc:288] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:14.937917 9755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:14.938234 9755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [term 1 LEADER]: Becoming Leader. State: Replica: ab3e6d1949f640549ca92550efeb00c1, State: Running, Role: LEADER
I20260619 06:57:14.938593 9755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } }
I20260619 06:57:14.938977 9750 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:14.940563 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3e6d1949f640549ca92550efeb00c1. Latest consensus state: current_term: 1 leader_uuid: "ab3e6d1949f640549ca92550efeb00c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } } }
I20260619 06:57:14.940707 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:14.941174 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3e6d1949f640549ca92550efeb00c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3e6d1949f640549ca92550efeb00c1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42555 } } }
I20260619 06:57:14.941282 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1 [sys.catalog]: This master's current role is: LEADER
W20260619 06:57:14.944473 9770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:14.944561 9770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:14.944626 9767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:14.945350 9767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:14.951776 9767 catalog_manager.cc:1347] Generated new cluster ID: 3ffd03d4c7c442c2bd4a1df63278c7a7
I20260619 06:57:14.951870 9767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:14.974449 9767 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:14.975698 9767 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:14.981328 9767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: Generated new TSK 0
I20260619 06:57:14.982136 9767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
Entry for principal kudu/127.22.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab.
I20260619 06:57:15.018790 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:15.143208 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:15.143536 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:15.143587 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:15.147961 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:15.148025 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:15.148124 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:15.153818 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.1.keytab
--principal=kudu/127.22.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:15.155251 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:15.156646 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:15.167613 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.167603 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.167668 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:15.168026 9776 server_base.cc:1034] running on GCE node
I20260619 06:57:15.168592 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:15.169394 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:15.170594 9776 hybrid_clock.cc:648] HybridClock initialized: now 1781852235170560 us; error 50 us; skew 500 ppm
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:15.173681 9776 init.cc:377] Logged in from keytab as kudu/127.22.113.1@KRBTEST.COM (short username kudu)
I20260619 06:57:15.174719 9776 webserver.cc:460] Webserver started at http://127.22.113.1:34031/ using document root <none> and password file <none>
I20260619 06:57:15.175441 9776 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:15.175539 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:15.175803 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:15.178131 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "fea2dcdac7f2434ea20bea377b4604b5"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "4f35e666f016e33be06fb20734d72594"
server_key_iv: "a4a818eb4e987ccee3ac51d9a02f0b6e"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.178747 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "fea2dcdac7f2434ea20bea377b4604b5"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "4f35e666f016e33be06fb20734d72594"
server_key_iv: "a4a818eb4e987ccee3ac51d9a02f0b6e"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.182992 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:57:15.185796 9791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.187181 9776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:15.187354 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "fea2dcdac7f2434ea20bea377b4604b5"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "4f35e666f016e33be06fb20734d72594"
server_key_iv: "a4a818eb4e987ccee3ac51d9a02f0b6e"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.187456 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:15.198530 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:15.201772 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:15.201999 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:15.202656 9776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:15.203657 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:15.203709 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.203753 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:15.203804 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.215746 9776 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:46257
I20260619 06:57:15.215824 9904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:46257 every 8 connection(s)
I20260619 06:57:15.216899 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:15.226429 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9776
I20260619 06:57:15.226559 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:15.226948 22980 external_mini_cluster.cc:1428] Setting key 651fcc4cda3cc911ca45982d1efd0fbe
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.1@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:15.232535 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.219010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:40013 (local address 127.22.113.62:42555)
0619 06:57:15.219156 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.219160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.220170 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 06:57:15.221638 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.221641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.221696 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.221779 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:15.223037 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.223650 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.224879 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.225119 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.225240 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.228057 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:15.228061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:15.228070 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:15.228095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:15.229821 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.230507 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.230512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.230516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.230645 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.231052 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.231057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.231061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.231390 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:15.231512 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.232095 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.232265 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:57:15.233305 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.219491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42555 (local address 127.22.113.1:40013)
0619 06:57:15.219989 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.220044 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.221412 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.221980 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.221988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.222364 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:15.222857 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.222871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.223771 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.223775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.224749 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.224759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.225282 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.225286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.225448 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.226085 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:15.226101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:15.227864 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:15.229957 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.229967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.229986 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.230369 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.230782 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.230784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.230786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.230932 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.231556 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:15.231561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:15.231826 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.232013 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.232302 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260619 06:57:15.234673 9905 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42555
I20260619 06:57:15.234961 9905 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:15.235523 9905 heartbeater.cc:511] Master 127.22.113.62:42555 requested a full tablet report, sending...
I20260619 06:57:15.237097 9714 ts_manager.cc:194] Registered new tserver with Master: fea2dcdac7f2434ea20bea377b4604b5 (127.22.113.1:46257)
WARNING: no policy specified for kudu/127.22.113.2@KRBTEST.COM; defaulting to no policy
I20260619 06:57:15.238397 9714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.1@KRBTEST.COM'} at 127.22.113.1:40013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
Entry for principal kudu/127.22.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab.
I20260619 06:57:15.259538 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:15.382915 9910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:15.383301 9910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:15.383383 9910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:15.387614 9910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:15.387704 9910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:15.387794 9910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:15.393455 9910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.2.keytab
--principal=kudu/127.22.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:15.394788 9910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:15.396109 9910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:15.406881 9916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.406886 9918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.407085 9915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:15.407624 9910 server_base.cc:1034] running on GCE node
I20260619 06:57:15.408098 9910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:15.408855 9910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:15.410061 9910 hybrid_clock.cc:648] HybridClock initialized: now 1781852235410017 us; error 62 us; skew 500 ppm
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:15.413257 9910 init.cc:377] Logged in from keytab as kudu/127.22.113.2@KRBTEST.COM (short username kudu)
I20260619 06:57:15.414409 9910 webserver.cc:460] Webserver started at http://127.22.113.2:34633/ using document root <none> and password file <none>
I20260619 06:57:15.415110 9910 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:15.415220 9910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:15.415551 9910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:15.418035 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "a47cdfbf7fb744bd949645059a1bb2aa"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "0611741ec2198af1cecef7f4bb0c6bab"
server_key_iv: "6b902d6b3bb584afb59af10616b3d9c7"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.418721 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "a47cdfbf7fb744bd949645059a1bb2aa"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "0611741ec2198af1cecef7f4bb0c6bab"
server_key_iv: "6b902d6b3bb584afb59af10616b3d9c7"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.423231 9910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 06:57:15.426095 9925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.427662 9910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:57:15.427819 9910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "a47cdfbf7fb744bd949645059a1bb2aa"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "0611741ec2198af1cecef7f4bb0c6bab"
server_key_iv: "6b902d6b3bb584afb59af10616b3d9c7"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.427937 9910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:15.449088 9910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:15.452548 9910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:15.452756 9910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:15.453390 9910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:15.454277 9910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:15.454350 9910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.454421 9910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:15.454476 9910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.464321 9910 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:44721
I20260619 06:57:15.464347 10038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:44721 every 8 connection(s)
I20260619 06:57:15.465262 9910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:15.467406 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 9910
I20260619 06:57:15.467535 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:15.467868 22980 external_mini_cluster.cc:1428] Setting key 2c3b5e34e833a0dbe4e4ddde91264181
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.2@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.3@KRBTEST.COM" created.
I20260619 06:57:15.481596 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.467659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:47807 (local address 127.22.113.62:42555)
0619 06:57:15.467840 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.467848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.469358 (+ 1510us) server_negotiation.cc:407] Connection header received
0619 06:57:15.470582 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.470585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.470643 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.470716 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:15.472141 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.472724 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.473941 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.474189 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.474300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.477568 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:15.477572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:15.477574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:15.477600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:15.478924 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.479637 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.479640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.479642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.479719 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.480088 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.480091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.480095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.480354 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:15.480483 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.481184 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.481372 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:57:15.482409 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.468080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42555 (local address 127.22.113.2:47807)
0619 06:57:15.469169 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.469225 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.470339 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.470918 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.470926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.471427 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:15.471998 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.472012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.472849 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.472853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.473809 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.473817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.474346 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.474350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.474549 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.475403 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:15.475420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:15.477325 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:15.479079 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.479085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.479097 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.479515 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.479813 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.479816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.479817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.479953 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.480500 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:15.480510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:15.480849 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.481063 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.481452 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":672,"threads_started":1}
I20260619 06:57:15.484007 10039 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42555
I20260619 06:57:15.484293 10039 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:15.484884 10039 heartbeater.cc:511] Master 127.22.113.62:42555 requested a full tablet report, sending...
I20260619 06:57:15.486187 9714 ts_manager.cc:194] Registered new tserver with Master: a47cdfbf7fb744bd949645059a1bb2aa (127.22.113.2:44721)
I20260619 06:57:15.486866 9714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.2@KRBTEST.COM'} at 127.22.113.2:47807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
Entry for principal kudu/127.22.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab.
I20260619 06:57:15.498571 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:15.622289 10044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:15.622653 10044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:15.622742 10044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:15.627125 10044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:15.627223 10044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:15.627351 10044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:15.633104 10044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.3.keytab
--principal=kudu/127.22.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:15.634408 10044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:15.635636 10044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:15.646914 10050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.646862 10052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:15.646867 10049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:15.647390 10044 server_base.cc:1034] running on GCE node
I20260619 06:57:15.647941 10044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:15.648671 10044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:15.649873 10044 hybrid_clock.cc:648] HybridClock initialized: now 1781852235649830 us; error 39 us; skew 500 ppm
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:15.653069 10044 init.cc:377] Logged in from keytab as kudu/127.22.113.3@KRBTEST.COM (short username kudu)
I20260619 06:57:15.654004 10044 webserver.cc:460] Webserver started at http://127.22.113.3:40657/ using document root <none> and password file <none>
I20260619 06:57:15.654615 10044 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:15.654685 10044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:15.654865 10044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:15.657267 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "5130a82fda06f867bd53ababe0bcb884"
server_key_iv: "6f7755b9d2569d03d0150fa18e1b8b5b"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.657842 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "5130a82fda06f867bd53ababe0bcb884"
server_key_iv: "6f7755b9d2569d03d0150fa18e1b8b5b"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.662022 10044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 06:57:15.664880 10059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.666482 10044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:15.666642 10044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2"
format_stamp: "Formatted at 2026-06-19 06:57:15 on dist-test-slave-g8zg"
server_key: "5130a82fda06f867bd53ababe0bcb884"
server_key_iv: "6f7755b9d2569d03d0150fa18e1b8b5b"
server_key_version: "encryptionkey@0"
I20260619 06:57:15.666752 10044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:15.694339 10044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:15.697721 10044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:15.697898 10044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:15.698520 10044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:15.699535 10044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:15.699594 10044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.699672 10044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:15.699723 10044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:15.710709 10044 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:40659
I20260619 06:57:15.710808 10172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:40659 every 8 connection(s)
I20260619 06:57:15.711804 10044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:15.716327 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10044
I20260619 06:57:15.716512 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:15.716831 22980 external_mini_cluster.cc:1428] Setting key 7b1a8205f02cd24d97798181ca9692ae
Jun 19 06:57:15 dist-test-slave-g8zg krb5kdc[9669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852235, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.3@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:15.728236 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.714134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:46355 (local address 127.22.113.62:42555)
0619 06:57:15.714359 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.714364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.715283 (+ 919us) server_negotiation.cc:407] Connection header received
0619 06:57:15.716946 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.716950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.717018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.717146 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:15.718912 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.719684 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.721053 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.721293 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.721393 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.724410 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:15.724413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:15.724416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:15.724443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:15.726068 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.726626 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.726629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.726630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.726705 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.727070 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.727073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.727074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.727349 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:15.727448 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.727999 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.728110 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:57:15.729342 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.714516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:42555 (local address 127.22.113.3:46355)
0619 06:57:15.715051 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.715103 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.716685 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.717361 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.717369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.717927 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:15.718725 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.718746 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.719825 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.719828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.720913 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.720923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.721452 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.721455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.721618 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.722383 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:15.722403 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:15.724217 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:15.726187 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.726192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.726207 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.726499 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.726795 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:15.726798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:15.726799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:15.726952 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:15.727459 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:15.727464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:15.727746 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.727937 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.728246 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260619 06:57:15.730796 10173 heartbeater.cc:347] Connected to a master server at 127.22.113.62:42555
I20260619 06:57:15.731120 10173 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:15.731760 10173 heartbeater.cc:511] Master 127.22.113.62:42555 requested a full tablet report, sending...
I20260619 06:57:15.733098 9714 ts_manager.cc:194] Registered new tserver with Master: 5aed5e88de7b4127b0a0a3b61b5aabb2 (127.22.113.3:40659)
I20260619 06:57:15.733950 9714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.113.3@KRBTEST.COM'} at 127.22.113.3:46355
I20260619 06:57:15.743643 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:15.755715 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.745948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.22.113.62:42555)
0619 06:57:15.746122 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.746127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.746324 (+ 197us) server_negotiation.cc:407] Connection header received
0619 06:57:15.746485 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.746488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.746551 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.746642 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:15.747407 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.747934 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.748798 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.749019 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.749124 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.751012 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:15.751019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:15.751024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:15.751057 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:15.753212 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.753915 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.753921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.753925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.754035 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:15.754511 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:15.754516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:15.754519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:15.754847 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:15.754995 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.755408 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.755538 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:57:15.759935 9714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:15.763656 9714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:15.780849 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.773720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:44721 (local address 127.0.0.1:33574)
0619 06:57:15.774729 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.774748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.774853 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.775761 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.775767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.775792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:15.775979 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.775987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.777369 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.777377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.778869 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.778878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780164 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.780288 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780324 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.780460 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.780612 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":4,"thread_start_us":149,"threads_started":1}
I20260619 06:57:15.781073 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.774960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.22.113.3:40659)
0619 06:57:15.776238 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.776270 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.776292 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:57:15.776376 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.776383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.776556 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.776707 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:15.776990 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.778044 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.779485 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780059 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780195 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780351 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.780802 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.780912 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":159,"threads_started":1}
I20260619 06:57:15.781118 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.774536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46257 (local address 127.0.0.1:60408)
0619 06:57:15.775057 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.775072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.775152 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.775975 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.775978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.775995 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:15.776158 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.776169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.777626 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.777630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.779062 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.779072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780578 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.780702 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780731 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.780814 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.780934 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":81,"thread_start_us":116,"threads_started":1}
I20260619 06:57:15.781299 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.773717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:40659 (local address 127.0.0.1:33352)
0619 06:57:15.774182 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:15.774229 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:15.774350 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:15.776720 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:15.776724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:15.776745 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:15.776859 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.776865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.778195 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.778198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.779371 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:15.779379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780412 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:15.780514 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780533 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:57:15.780596 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:57:15.780657 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260619 06:57:15.781740 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.774780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.22.113.1:46257)
0619 06:57:15.775453 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.775462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.775488 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:57:15.775565 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.775571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.775807 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.775964 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:15.776313 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.777482 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.779280 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780192 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780494 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.781399 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.781547 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":68,"threads_started":1}
I20260619 06:57:15.780864 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:15.773895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.22.113.2:44721)
0619 06:57:15.774877 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:15.774895 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:15.774924 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:57:15.775022 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:15.775029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:15.775544 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:15.775772 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:15.776118 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.777228 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.779007 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:15.780001 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:15.780145 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:15.780346 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:15.780513 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 06:57:15.780640 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":136,"threads_started":1}
I20260619 06:57:15.784229 10107 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ace64488c484e973542e6d0f699bf (DEFAULT_TABLE table=test-table [id=e728601a93c44513916faff4417d73b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:15.785226 10107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ace64488c484e973542e6d0f699bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:15.785120 9839 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ace64488c484e973542e6d0f699bf (DEFAULT_TABLE table=test-table [id=e728601a93c44513916faff4417d73b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:15.785122 9973 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ace64488c484e973542e6d0f699bf (DEFAULT_TABLE table=test-table [id=e728601a93c44513916faff4417d73b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:15.786332 9839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ace64488c484e973542e6d0f699bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:15.786332 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ace64488c484e973542e6d0f699bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:15.791687 10190 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Bootstrap starting.
I20260619 06:57:15.794008 10190 tablet_bootstrap.cc:654] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:15.795307 10190 log.cc:824] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:15.795957 10192 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Bootstrap starting.
I20260619 06:57:15.795957 10191 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Bootstrap starting.
I20260619 06:57:15.797492 10190 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: No bootstrap required, opened a new log
I20260619 06:57:15.797708 10190 ts_tablet_manager.cc:1397] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:57:15.798830 10192 tablet_bootstrap.cc:654] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:15.798830 10191 tablet_bootstrap.cc:654] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:15.799997 10192 log.cc:824] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:15.800225 10191 log.cc:824] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:15.800799 10190 raft_consensus.cc:348] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.801074 10190 raft_consensus.cc:374] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:15.801123 10190 raft_consensus.cc:729] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aed5e88de7b4127b0a0a3b61b5aabb2, State: Initialized, Role: FOLLOWER
I20260619 06:57:15.801615 10190 consensus_queue.cc:260] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.802474 10173 heartbeater.cc:503] Master 127.22.113.62:42555 was elected leader, sending a full tablet report...
I20260619 06:57:15.802508 10191 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: No bootstrap required, opened a new log
I20260619 06:57:15.802574 10192 tablet_bootstrap.cc:492] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: No bootstrap required, opened a new log
I20260619 06:57:15.802672 10190 ts_tablet_manager.cc:1428] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:57:15.802709 10191 ts_tablet_manager.cc:1397] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:57:15.802783 10192 ts_tablet_manager.cc:1397] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:57:15.806047 10191 raft_consensus.cc:348] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.806320 10191 raft_consensus.cc:374] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:15.806372 10191 raft_consensus.cc:729] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47cdfbf7fb744bd949645059a1bb2aa, State: Initialized, Role: FOLLOWER
I20260619 06:57:15.806804 10191 consensus_queue.cc:260] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.807561 10039 heartbeater.cc:503] Master 127.22.113.62:42555 was elected leader, sending a full tablet report...
I20260619 06:57:15.807828 10191 ts_tablet_manager.cc:1428] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:57:15.808094 10192 raft_consensus.cc:348] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.808454 10192 raft_consensus.cc:374] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:15.808524 10192 raft_consensus.cc:729] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea2dcdac7f2434ea20bea377b4604b5, State: Initialized, Role: FOLLOWER
I20260619 06:57:15.809090 10192 consensus_queue.cc:260] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:15.809928 9905 heartbeater.cc:503] Master 127.22.113.62:42555 was elected leader, sending a full tablet report...
I20260619 06:57:15.810350 10192 ts_tablet_manager.cc:1428] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260619 06:57:15.963052 10174 tablet.cc:2369] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:57:15.966797 10040 tablet.cc:2369] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:57:15.968466 9906 tablet.cc:2369] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:16.004614 10196 raft_consensus.cc:482] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:16.004856 10196 raft_consensus.cc:504] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:16.006040 10196 leader_election.cc:288] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47cdfbf7fb744bd949645059a1bb2aa (127.22.113.2:44721), fea2dcdac7f2434ea20bea377b4604b5 (127.22.113.1:46257)
I20260619 06:57:16.010761 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.006374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46257 (local address 127.22.113.3:59585)
0619 06:57:16.006506 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:16.006524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:16.006632 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:16.007053 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:16.007057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:16.007081 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:16.007327 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.007338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.008143 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.008148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.009323 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.009333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.010274 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.010276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.010364 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.010419 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:57:16.010509 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:57:16.010597 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":51}
I20260619 06:57:16.010825 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.006569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:59585 (local address 127.22.113.1:46257)
0619 06:57:16.006735 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.006741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.006762 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:57:16.006848 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.006854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.006922 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.007035 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:16.007456 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.008022 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.009454 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.010128 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.010276 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.010343 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.010551 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.010640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:57:16.010959 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.006740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:35021 (local address 127.22.113.2:44721)
0619 06:57:16.006880 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.006885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.007026 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:57:16.007207 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.007210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.007314 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.007411 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:16.007739 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.008260 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.009646 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.010318 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.010448 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.010522 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.010744 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.010829 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:57:16.010960 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.006475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:44721 (local address 127.22.113.3:35021)
0619 06:57:16.006921 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:16.006937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:16.007068 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:16.007424 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:16.007426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:16.007444 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:16.007613 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.007619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.008392 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.008396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.009496 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.009507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.010449 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.010455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.010605 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.010625 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:57:16.010702 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:57:16.010753 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260619 06:57:16.011620 9859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ace64488c484e973542e6d0f699bf" candidate_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea2dcdac7f2434ea20bea377b4604b5" is_pre_election: true
I20260619 06:57:16.011879 9993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ace64488c484e973542e6d0f699bf" candidate_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" is_pre_election: true
I20260619 06:57:16.011948 9859 raft_consensus.cc:2454] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aed5e88de7b4127b0a0a3b61b5aabb2 in term 0.
I20260619 06:57:16.012293 9993 raft_consensus.cc:2454] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aed5e88de7b4127b0a0a3b61b5aabb2 in term 0.
I20260619 06:57:16.012506 10063 leader_election.cc:302] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aed5e88de7b4127b0a0a3b61b5aabb2, fea2dcdac7f2434ea20bea377b4604b5; no voters:
I20260619 06:57:16.012787 10196 raft_consensus.cc:2781] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:16.012873 10196 raft_consensus.cc:482] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:16.012925 10196 raft_consensus.cc:3037] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:16.013944 10196 raft_consensus.cc:504] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:16.014335 10196 leader_election.cc:288] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [CANDIDATE]: Term 1 election: Requested vote from peers a47cdfbf7fb744bd949645059a1bb2aa (127.22.113.2:44721), fea2dcdac7f2434ea20bea377b4604b5 (127.22.113.1:46257)
I20260619 06:57:16.014724 9859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ace64488c484e973542e6d0f699bf" candidate_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea2dcdac7f2434ea20bea377b4604b5"
I20260619 06:57:16.014786 9993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ace64488c484e973542e6d0f699bf" candidate_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47cdfbf7fb744bd949645059a1bb2aa"
I20260619 06:57:16.014865 9859 raft_consensus.cc:3037] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:16.014950 9993 raft_consensus.cc:3037] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:16.016155 9859 raft_consensus.cc:2454] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aed5e88de7b4127b0a0a3b61b5aabb2 in term 1.
I20260619 06:57:16.016378 9993 raft_consensus.cc:2454] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aed5e88de7b4127b0a0a3b61b5aabb2 in term 1.
I20260619 06:57:16.016467 10063 leader_election.cc:302] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aed5e88de7b4127b0a0a3b61b5aabb2, fea2dcdac7f2434ea20bea377b4604b5; no voters:
I20260619 06:57:16.016649 10196 raft_consensus.cc:2781] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:16.016911 10196 raft_consensus.cc:686] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 1 LEADER]: Becoming Leader. State: Replica: 5aed5e88de7b4127b0a0a3b61b5aabb2, State: Running, Role: LEADER
I20260619 06:57:16.017282 10196 consensus_queue.cc:237] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } }
I20260619 06:57:16.020473 9709 catalog_manager.cc:5495] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aed5e88de7b4127b0a0a3b61b5aabb2 (127.22.113.3). New cstate: current_term: 1 leader_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aed5e88de7b4127b0a0a3b61b5aabb2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 40659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 } health_report { overall_health: UNKNOWN } } }
I20260619 06:57:16.076057 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.072377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.22.113.3:40659)
0619 06:57:16.072567 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.072572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.072584 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:57:16.072656 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.072658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.072712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.072826 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:16.073064 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.073668 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.074517 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.074796 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.074897 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.075192 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:16.075633 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:16.075764 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.075802 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.075871 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:57:16.082839 9859 raft_consensus.cc:1261] T 6b5ace64488c484e973542e6d0f699bf P fea2dcdac7f2434ea20bea377b4604b5 [term 1 FOLLOWER]: Refusing update from remote peer 5aed5e88de7b4127b0a0a3b61b5aabb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:16.082888 9993 raft_consensus.cc:1261] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 1 FOLLOWER]: Refusing update from remote peer 5aed5e88de7b4127b0a0a3b61b5aabb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:16.083721 10200 consensus_queue.cc:1035] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea2dcdac7f2434ea20bea377b4604b5" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:16.083989 10196 consensus_queue.cc:1035] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47cdfbf7fb744bd949645059a1bb2aa" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:16.094321 10204 mvcc.cc:205] Tried to move back new op lower bound from 7298466758987079680 to 7298466758729256960. Current Snapshot: MvccSnapshot[applied={T|T < 7298466758987079680}]
I20260619 06:57:16.104456 9714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:16.104671 9714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:16.107426 9714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab3e6d1949f640549ca92550efeb00c1: Sending DeleteTablet for 3 replicas of tablet 6b5ace64488c484e973542e6d0f699bf
I20260619 06:57:16.108309 10107 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ace64488c484e973542e6d0f699bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:16 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:33352
I20260619 06:57:16.108309 9973 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ace64488c484e973542e6d0f699bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:16 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:33574
I20260619 06:57:16.108887 10213 tablet_replica.cc:330] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: stopping tablet replica
I20260619 06:57:16.108929 9839 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ace64488c484e973542e6d0f699bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:16 UTC) from {username='kudu', principal='kudu/127.22.113.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260619 06:57:16.109236 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9776
I20260619 06:57:16.109277 10213 raft_consensus.cc:2229] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:16.109644 10213 raft_consensus.cc:2258] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:16.110010 10214 tablet_replica.cc:330] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: stopping tablet replica
I20260619 06:57:16.110333 10214 raft_consensus.cc:2229] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:57:16.110809 10214 raft_consensus.cc:2258] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:16.111474 10213 ts_tablet_manager.cc:1905] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:16.112282 10214 ts_tablet_manager.cc:1905] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:16.114775 10213 ts_tablet_manager.cc:1918] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:16.114886 10213 log.cc:1196] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/6b5ace64488c484e973542e6d0f699bf
I20260619 06:57:16.115216 10213 ts_tablet_manager.cc:1939] T 6b5ace64488c484e973542e6d0f699bf P a47cdfbf7fb744bd949645059a1bb2aa: Deleting consensus metadata
I20260619 06:57:16.115727 10214 ts_tablet_manager.cc:1918] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:16.115883 10214 log.cc:1196] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/6b5ace64488c484e973542e6d0f699bf
I20260619 06:57:16.116252 10214 ts_tablet_manager.cc:1939] T 6b5ace64488c484e973542e6d0f699bf P 5aed5e88de7b4127b0a0a3b61b5aabb2: Deleting consensus metadata
I20260619 06:57:16.116747 9700 catalog_manager.cc:4841] TS a47cdfbf7fb744bd949645059a1bb2aa (127.22.113.2:44721): tablet 6b5ace64488c484e973542e6d0f699bf (table test-table [id=e728601a93c44513916faff4417d73b6]) successfully deleted
I20260619 06:57:16.117404 9701 catalog_manager.cc:4841] TS 5aed5e88de7b4127b0a0a3b61b5aabb2 (127.22.113.3:40659): tablet 6b5ace64488c484e973542e6d0f699bf (table test-table [id=e728601a93c44513916faff4417d73b6]) successfully deleted
W20260619 06:57:16.119961 9701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:46257 (error 108)
I20260619 06:57:16.120106 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9910
I20260619 06:57:16.122997 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.121208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:46257 (local address 127.0.0.1:60412)
0619 06:57:16.122840 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:16.122907 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1537}
W20260619 06:57:16.123193 9701 catalog_manager.cc:4584] TS fea2dcdac7f2434ea20bea377b4604b5 (127.22.113.1:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b5ace64488c484e973542e6d0f699bf: Network error: Client connection negotiation failed: client connection to 127.22.113.1:46257: connect: Connection refused (error 111)
I20260619 06:57:16.127646 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10044
I20260619 06:57:16.135403 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 9682
2026-06-19T06:57:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2582 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T06:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:16Z Disabled control of system clock
I20260619 06:57:16.167745 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34279
--webserver_interface=127.22.113.62
--webserver_port=0
--builtin_ntp_servers=127.22.113.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:16.284687 10220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:16.284965 10220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:16.285002 10220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:16.289067 10220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:16.289129 10220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:16.289145 10220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:16.289163 10220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:16.289211 10220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:16.294129 10220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:34279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:16.295291 10220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:16.296720 10220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:16.305744 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:16.305801 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:16.305824 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:16.306160 10220 server_base.cc:1034] running on GCE node
I20260619 06:57:16.306746 10220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:16.307986 10220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:16.309275 10220 hybrid_clock.cc:648] HybridClock initialized: now 1781852236309233 us; error 62 us; skew 500 ppm
I20260619 06:57:16.311209 10220 webserver.cc:460] Webserver started at http://127.22.113.62:46111/ using document root <none> and password file <none>
I20260619 06:57:16.311826 10220 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:16.311918 10220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:16.312129 10220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:16.314162 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "cd42ef09e401440d8cbf20977bb565fc"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "bd8ac9fb4eccc92d99a19d14ae3085aa"
server_key_iv: "f8414e10ed91f9e6b1cb20368d22a831"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.314726 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "cd42ef09e401440d8cbf20977bb565fc"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "bd8ac9fb4eccc92d99a19d14ae3085aa"
server_key_iv: "f8414e10ed91f9e6b1cb20368d22a831"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.318651 10220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:57:16.321467 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.322999 10220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:16.323179 10220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "cd42ef09e401440d8cbf20977bb565fc"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "bd8ac9fb4eccc92d99a19d14ae3085aa"
server_key_iv: "f8414e10ed91f9e6b1cb20368d22a831"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.323346 10220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:16.354650 10220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:16.355489 10220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:16.355718 10220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:16.363523 10220 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:34279
I20260619 06:57:16.363536 10286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:34279 every 8 connection(s)
I20260619 06:57:16.364481 10220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:16.364845 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10220
I20260619 06:57:16.365000 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:16.365329 22980 external_mini_cluster.cc:1428] Setting key 97a0e3d164e6e307b38bb73e841aaf80
I20260619 06:57:16.369169 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:16.377097 10287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: Bootstrap starting.
I20260619 06:57:16.377264 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.367032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.22.113.62:34279)
0619 06:57:16.367896 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.367920 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.367975 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:57:16.368991 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.369020 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.369572 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.370010 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:16.370503 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.372074 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.373026 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.373738 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.373935 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.374896 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:16.374904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:16.374922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:16.375506 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:16.375588 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:16.375621 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:16.375734 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.375931 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.376233 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":204,"threads_started":1}
I20260619 06:57:16.379992 10287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:16.380841 10287 log.cc:824] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:16.383044 10287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: No bootstrap required, opened a new log
I20260619 06:57:16.387173 10287 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } }
I20260619 06:57:16.387478 10287 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:16.387521 10287 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd42ef09e401440d8cbf20977bb565fc, State: Initialized, Role: FOLLOWER
I20260619 06:57:16.387921 10287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } }
I20260619 06:57:16.388036 10287 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:16.388077 10287 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:16.388157 10287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:16.389159 10287 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } }
I20260619 06:57:16.389482 10287 leader_election.cc:302] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd42ef09e401440d8cbf20977bb565fc; no voters:
I20260619 06:57:16.389807 10287 leader_election.cc:288] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:16.390084 10292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:16.390564 10292 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [term 1 LEADER]: Becoming Leader. State: Replica: cd42ef09e401440d8cbf20977bb565fc, State: Running, Role: LEADER
I20260619 06:57:16.390995 10292 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } }
I20260619 06:57:16.391312 10287 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:16.392918 10293 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd42ef09e401440d8cbf20977bb565fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } } }
I20260619 06:57:16.393065 10293 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:16.393294 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd42ef09e401440d8cbf20977bb565fc. Latest consensus state: current_term: 1 leader_uuid: "cd42ef09e401440d8cbf20977bb565fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd42ef09e401440d8cbf20977bb565fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 34279 } } }
I20260619 06:57:16.393399 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:16.393600 10301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:16.396800 10301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:16.402570 10301 catalog_manager.cc:1347] Generated new cluster ID: 2daf6338ee1b4c2e9055d85331575504
I20260619 06:57:16.402638 10301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:16.422391 10301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:16.423902 10301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:16.433216 10301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: Generated new TSK 0
I20260619 06:57:16.434046 10301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:16.440092 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.1:0
--local_ip_for_outbound_sockets=127.22.113.1
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--builtin_ntp_servers=127.22.113.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:57:16.564329 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:16.564831 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:16.564936 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:16.569094 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:16.569150 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:16.569208 10311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.1
I20260619 06:57:16.574069 10311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.113.1
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.1
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:16.575453 10311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:16.577101 10311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:16.588210 10317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:16.588264 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:16.588415 10311 server_base.cc:1034] running on GCE node
W20260619 06:57:16.588264 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:16.589031 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:16.589813 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:16.590986 10311 hybrid_clock.cc:648] HybridClock initialized: now 1781852236590966 us; error 35 us; skew 500 ppm
I20260619 06:57:16.593223 10311 webserver.cc:460] Webserver started at http://127.22.113.1:43689/ using document root <none> and password file <none>
I20260619 06:57:16.593902 10311 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:16.593969 10311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:16.594203 10311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:16.596555 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data/instance:
uuid: "0426708f452849fe84e9214e655a0c39"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "68d6521e1fcc5f1167bf3f45390b2049"
server_key_iv: "dda50a6a67ec143e13a7a88fe55133cf"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.597183 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance:
uuid: "0426708f452849fe84e9214e655a0c39"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "68d6521e1fcc5f1167bf3f45390b2049"
server_key_iv: "dda50a6a67ec143e13a7a88fe55133cf"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.601634 10311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:16.604671 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.606227 10311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 06:57:16.606380 10311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal
uuid: "0426708f452849fe84e9214e655a0c39"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "68d6521e1fcc5f1167bf3f45390b2049"
server_key_iv: "dda50a6a67ec143e13a7a88fe55133cf"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.606513 10311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:16.633338 10311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:16.634174 10311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:16.634392 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:16.635092 10311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:16.636221 10311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:16.636296 10311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.636373 10311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:16.636425 10311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.647226 10311 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.1:36443
I20260619 06:57:16.647277 10438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.1:36443 every 8 connection(s)
I20260619 06:57:16.648288 10311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:16.658085 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10311
I20260619 06:57:16.658238 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:16.658537 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.650333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.1:50859 (local address 127.22.113.62:34279)
0619 06:57:16.650544 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.650550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.651853 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 06:57:16.653463 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.653466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.653531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.653621 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:16.654543 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.655113 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.656327 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.656562 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.656666 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.657850 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:16.657853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:16.657855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:16.657868 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:16.657887 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:16.657895 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:16.658014 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.658303 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.658399 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:57:16.658588 22980 external_mini_cluster.cc:1428] Setting key 42fc783435e6753b4d95156f13210a63
I20260619 06:57:16.659507 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.650864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34279 (local address 127.22.113.1:50859)
0619 06:57:16.651675 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:16.651724 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:16.653245 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:16.653793 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:16.653799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:16.653882 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:16.654413 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.654425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.655230 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.655232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.656177 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.656186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.656724 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.656727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.656871 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.657577 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:16.657578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:16.657619 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:16.657680 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:16.658041 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:16.658050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:16.658057 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:57:16.658210 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 06:57:16.658511 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260619 06:57:16.660948 10439 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34279
I20260619 06:57:16.661078 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.2:0
--local_ip_for_outbound_sockets=127.22.113.2
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--builtin_ntp_servers=127.22.113.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:16.661239 10439 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:16.661861 10439 heartbeater.cc:511] Master 127.22.113.62:34279 requested a full tablet report, sending...
I20260619 06:57:16.663757 10251 ts_manager.cc:194] Registered new tserver with Master: 0426708f452849fe84e9214e655a0c39 (127.22.113.1:36443)
I20260619 06:57:16.665544 10251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.1:50859
W20260619 06:57:16.789925 10442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:16.790243 10442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:16.790288 10442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:16.794996 10442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:16.795083 10442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:16.795151 10442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.2
I20260619 06:57:16.800251 10442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.113.2
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.2
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:16.801614 10442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:16.803284 10442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:16.814535 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:16.814539 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:16.814535 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:16.815016 10442 server_base.cc:1034] running on GCE node
I20260619 06:57:16.815548 10442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:16.816359 10442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:16.817548 10442 hybrid_clock.cc:648] HybridClock initialized: now 1781852236817528 us; error 35 us; skew 500 ppm
I20260619 06:57:16.819882 10442 webserver.cc:460] Webserver started at http://127.22.113.2:43489/ using document root <none> and password file <none>
I20260619 06:57:16.820567 10442 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:16.820637 10442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:16.820870 10442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:16.823032 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data/instance:
uuid: "5b57713a45c347088d151c953d500764"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "06e5c37359c80ece96acb312616fbe8a"
server_key_iv: "4bd1f7e35d7133787de8aeb3dbfee74c"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.823738 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance:
uuid: "5b57713a45c347088d151c953d500764"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "06e5c37359c80ece96acb312616fbe8a"
server_key_iv: "4bd1f7e35d7133787de8aeb3dbfee74c"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.827862 10442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:16.830592 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.832216 10442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:16.832381 10442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal
uuid: "5b57713a45c347088d151c953d500764"
format_stamp: "Formatted at 2026-06-19 06:57:16 on dist-test-slave-g8zg"
server_key: "06e5c37359c80ece96acb312616fbe8a"
server_key_iv: "4bd1f7e35d7133787de8aeb3dbfee74c"
server_key_version: "encryptionkey@0"
I20260619 06:57:16.832499 10442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:16.854573 10442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:16.855351 10442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:16.855561 10442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:16.856180 10442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:16.857178 10442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:16.857254 10442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.857326 10442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:16.857376 10442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:16.867347 10442 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.2:36457
I20260619 06:57:16.867432 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.2:36457 every 8 connection(s)
I20260619 06:57:16.868321 10442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:16.869669 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10442
I20260619 06:57:16.869809 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:16.870123 22980 external_mini_cluster.cc:1428] Setting key 2ccfe95973e224e4bc8699384b4594a0
I20260619 06:57:16.872304 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.113.3:0
--local_ip_for_outbound_sockets=127.22.113.3
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--builtin_ntp_servers=127.22.113.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:57:16.879894 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.870592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:41505 (local address 127.22.113.62:34279)
0619 06:57:16.870909 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:16.870915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:16.872231 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 06:57:16.873840 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:16.873845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:16.873912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:16.874026 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:16.875038 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.875960 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.877440 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.877867 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.878006 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.879231 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:16.879238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:16.879273 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:16.879292 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:16.879320 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:16.879335 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:16.879444 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:16.879595 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:57:16.879676 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260619 06:57:16.880800 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:16.871149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34279 (local address 127.22.113.2:41505)
0619 06:57:16.872044 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:16.872098 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:16.873610 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:16.874200 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:16.874207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:16.874324 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:16.874885 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.874897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.876115 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.876118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.877291 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:16.877301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:16.878041 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:16.878044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:16.878196 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:16.878945 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:16.878947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:16.878991 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:16.879060 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:16.879441 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:16.879447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:16.879451 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:57:16.879554 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:57:16.879809 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":187,"threads_started":1}
I20260619 06:57:16.882179 10570 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34279
I20260619 06:57:16.882551 10570 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:16.883378 10570 heartbeater.cc:511] Master 127.22.113.62:34279 requested a full tablet report, sending...
I20260619 06:57:16.884673 10251 ts_manager.cc:194] Registered new tserver with Master: 5b57713a45c347088d151c953d500764 (127.22.113.2:36457)
I20260619 06:57:16.885417 10251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.2:41505
W20260619 06:57:16.990535 10573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:16.990809 10573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:16.990849 10573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:16.995170 10573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:16.995234 10573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:16.995378 10573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.113.3
I20260619 06:57:17.000240 10573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.113.3
--webserver_port=0
--tserver_master_addrs=127.22.113.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.113.3
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:17.001401 10573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:17.002822 10573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:17.013314 10578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:17.013314 10581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:17.013386 10579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:17.013826 10573 server_base.cc:1034] running on GCE node
I20260619 06:57:17.014436 10573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:17.015309 10573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:17.016558 10573 hybrid_clock.cc:648] HybridClock initialized: now 1781852237016520 us; error 59 us; skew 500 ppm
I20260619 06:57:17.019070 10573 webserver.cc:460] Webserver started at http://127.22.113.3:33327/ using document root <none> and password file <none>
I20260619 06:57:17.019877 10573 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:17.019979 10573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:17.020270 10573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:17.022775 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data/instance:
uuid: "d4b1c9ecab364b20b1bc05b8946c2be2"
format_stamp: "Formatted at 2026-06-19 06:57:17 on dist-test-slave-g8zg"
server_key: "3b1c72f634404f1c2c44ebcece74ba3c"
server_key_iv: "68eaf5eb02536f26e52e5594971ab074"
server_key_version: "encryptionkey@0"
I20260619 06:57:17.023491 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b1c9ecab364b20b1bc05b8946c2be2"
format_stamp: "Formatted at 2026-06-19 06:57:17 on dist-test-slave-g8zg"
server_key: "3b1c72f634404f1c2c44ebcece74ba3c"
server_key_iv: "68eaf5eb02536f26e52e5594971ab074"
server_key_version: "encryptionkey@0"
I20260619 06:57:17.028154 10573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:57:17.031167 10587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:17.032606 10573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:57:17.032761 10573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal
uuid: "d4b1c9ecab364b20b1bc05b8946c2be2"
format_stamp: "Formatted at 2026-06-19 06:57:17 on dist-test-slave-g8zg"
server_key: "3b1c72f634404f1c2c44ebcece74ba3c"
server_key_iv: "68eaf5eb02536f26e52e5594971ab074"
server_key_version: "encryptionkey@0"
I20260619 06:57:17.032889 10573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:17.053893 10573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:17.054610 10573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:17.054802 10573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:17.055459 10573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:17.056391 10573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:17.056468 10573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:17.056535 10573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:17.056579 10573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:17.065927 10573 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.3:36693
I20260619 06:57:17.066035 10700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.3:36693 every 8 connection(s)
I20260619 06:57:17.067176 10573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:17.069365 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10573
I20260619 06:57:17.069478 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:17.069795 22980 external_mini_cluster.cc:1428] Setting key 113658dc1e6a6536066ec1e4e45e9016
I20260619 06:57:17.078215 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.069527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.3:40337 (local address 127.22.113.62:34279)
0619 06:57:17.069681 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.069685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.071231 (+ 1546us) server_negotiation.cc:407] Connection header received
0619 06:57:17.073068 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.073071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.073134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.073257 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:17.074284 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.074857 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.076110 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.076339 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.076452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.077670 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:17.077674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:17.077677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:17.077704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:17.077722 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:17.077730 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:17.077812 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.077983 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.078047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:57:17.079142 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.070045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:34279 (local address 127.22.113.3:40337)
0619 06:57:17.071046 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.071117 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.072830 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.073454 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.073462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.073555 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:17.074134 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.074147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.074987 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.074991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.075987 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.075996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.076498 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.076502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.076671 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.077396 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:57:17.077398 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:17.077441 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:57:17.077512 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:17.077812 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:17.077823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:17.077832 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.077973 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.078288 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":139,"threads_started":1}
I20260619 06:57:17.080394 10701 heartbeater.cc:347] Connected to a master server at 127.22.113.62:34279
I20260619 06:57:17.080624 10701 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:17.081169 10701 heartbeater.cc:511] Master 127.22.113.62:34279 requested a full tablet report, sending...
I20260619 06:57:17.082312 10251 ts_manager.cc:194] Registered new tserver with Master: d4b1c9ecab364b20b1bc05b8946c2be2 (127.22.113.3:36693)
I20260619 06:57:17.082893 10251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.113.3:40337
I20260619 06:57:17.084440 22980 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:17.090162 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.086416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.22.113.62:34279)
0619 06:57:17.086570 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.086575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.086784 (+ 209us) server_negotiation.cc:407] Connection header received
0619 06:57:17.086893 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.086895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.086934 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.087002 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:17.087332 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.087840 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.088667 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.088887 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.088968 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.089719 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:17.089722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:17.089724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:17.089735 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:17.089754 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:17.089774 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:17.089835 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.089916 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.089960 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:57:17.093636 10250 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:17.097097 10250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:17.112793 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.106711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.22.113.1:36443)
0619 06:57:17.107053 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.107061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.107619 (+ 558us) server_negotiation.cc:407] Connection header received
0619 06:57:17.107777 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.107785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.108030 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.108205 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:17.108750 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.109725 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.111528 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.112087 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.112213 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.112446 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.112503 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.112606 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260619 06:57:17.112795 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.106581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36443 (local address 127.0.0.1:44246)
0619 06:57:17.107195 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.107279 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.107428 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.108406 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.108415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.108448 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:17.108624 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.108631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.109895 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.109898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.111339 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.111349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.112211 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.112214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.112328 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.112368 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.112473 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.112617 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"spinlock_wait_cycles":896,"thread_start_us":99,"threads_started":1}
I20260619 06:57:17.114295 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.107525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.22.113.3:36693)
0619 06:57:17.107961 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.107969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.108235 (+ 266us) server_negotiation.cc:407] Connection header received
0619 06:57:17.108443 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.108449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.108677 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.108842 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:17.109376 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.110329 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.112217 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.113337 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.113413 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.113634 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.113988 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.114095 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":106,"threads_started":1}
I20260619 06:57:17.114480 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.107409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.2:36457 (local address 127.0.0.1:50944)
0619 06:57:17.108340 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.108356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.108459 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.109013 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.109016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.109028 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:17.109247 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.109254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.110640 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.110643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.111910 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.111920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.114039 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.114042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.114121 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.114140 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.114223 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.114308 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":116,"spinlock_wait_cycles":4992,"thread_start_us":110,"threads_started":1}
I20260619 06:57:17.114588 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.107886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.22.113.2:36457)
0619 06:57:17.108308 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.108324 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.108446 (+ 122us) server_negotiation.cc:407] Connection header received
0619 06:57:17.108593 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.108597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.108744 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.108867 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:17.109397 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.110517 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.112063 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.112940 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.113796 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.113973 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.114291 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.114415 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":165,"threads_started":1}
I20260619 06:57:17.114988 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.106830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:36693 (local address 127.0.0.1:38542)
0619 06:57:17.108137 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.108153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.108314 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.108996 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.108999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.109018 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:17.109247 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.109254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.110461 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.110466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.112100 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.112108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.113731 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.113735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.113829 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.113853 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.114823 (+ 970us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.114902 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":135,"thread_start_us":139,"threads_started":1}
I20260619 06:57:17.116261 10373 tablet_service.cc:1467] Processing CreateTablet for tablet ed3b52357b4e4f9a937fc2804834db19 (DEFAULT_TABLE table=test-table [id=cf1e4649c06243e69e976ee1be5d05b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:17.117581 10373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3b52357b4e4f9a937fc2804834db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:17.117669 10635 tablet_service.cc:1467] Processing CreateTablet for tablet ed3b52357b4e4f9a937fc2804834db19 (DEFAULT_TABLE table=test-table [id=cf1e4649c06243e69e976ee1be5d05b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:17.118592 10635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3b52357b4e4f9a937fc2804834db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:17.121658 10504 tablet_service.cc:1467] Processing CreateTablet for tablet ed3b52357b4e4f9a937fc2804834db19 (DEFAULT_TABLE table=test-table [id=cf1e4649c06243e69e976ee1be5d05b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:17.122617 10504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3b52357b4e4f9a937fc2804834db19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:17.126349 10718 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Bootstrap starting.
I20260619 06:57:17.128118 10719 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Bootstrap starting.
I20260619 06:57:17.129281 10718 tablet_bootstrap.cc:654] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:17.130044 10719 tablet_bootstrap.cc:654] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:17.130177 10718 log.cc:824] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:17.130370 10720 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Bootstrap starting.
I20260619 06:57:17.130760 10719 log.cc:824] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:17.132575 10720 tablet_bootstrap.cc:654] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:17.132838 10719 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: No bootstrap required, opened a new log
I20260619 06:57:17.132838 10718 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: No bootstrap required, opened a new log
I20260619 06:57:17.133036 10718 ts_tablet_manager.cc:1397] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:57:17.133036 10719 ts_tablet_manager.cc:1397] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:57:17.133481 10720 log.cc:824] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:17.135926 10720 tablet_bootstrap.cc:492] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: No bootstrap required, opened a new log
I20260619 06:57:17.136168 10720 ts_tablet_manager.cc:1397] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:57:17.137638 10718 raft_consensus.cc:348] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.137638 10719 raft_consensus.cc:348] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.137933 10718 raft_consensus.cc:374] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:17.137933 10719 raft_consensus.cc:374] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:17.138003 10719 raft_consensus.cc:729] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0426708f452849fe84e9214e655a0c39, State: Initialized, Role: FOLLOWER
I20260619 06:57:17.138002 10718 raft_consensus.cc:729] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b1c9ecab364b20b1bc05b8946c2be2, State: Initialized, Role: FOLLOWER
I20260619 06:57:17.138487 10719 consensus_queue.cc:260] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.138487 10718 consensus_queue.cc:260] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.139317 10439 heartbeater.cc:503] Master 127.22.113.62:34279 was elected leader, sending a full tablet report...
I20260619 06:57:17.139585 10719 ts_tablet_manager.cc:1428] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 06:57:17.139672 10701 heartbeater.cc:503] Master 127.22.113.62:34279 was elected leader, sending a full tablet report...
I20260619 06:57:17.140027 10718 ts_tablet_manager.cc:1428] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:57:17.141760 10720 raft_consensus.cc:348] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.142041 10720 raft_consensus.cc:374] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:17.142107 10720 raft_consensus.cc:729] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b57713a45c347088d151c953d500764, State: Initialized, Role: FOLLOWER
I20260619 06:57:17.142576 10720 consensus_queue.cc:260] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.143404 10570 heartbeater.cc:503] Master 127.22.113.62:34279 was elected leader, sending a full tablet report...
I20260619 06:57:17.143536 10720 ts_tablet_manager.cc:1428] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260619 06:57:17.149595 10440 tablet.cc:2369] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:17.301112 10726 raft_consensus.cc:482] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:17.301400 10726 raft_consensus.cc:504] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.302778 10726 leader_election.cc:288] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0426708f452849fe84e9214e655a0c39 (127.22.113.1:36443), d4b1c9ecab364b20b1bc05b8946c2be2 (127.22.113.3:36693)
I20260619 06:57:17.307938 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.303304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:39273 (local address 127.22.113.3:36693)
0619 06:57:17.303422 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.303426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.303447 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:57:17.303629 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.303632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.303691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.303774 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:17.304145 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.304884 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.306465 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.307128 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.307291 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.307341 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.307723 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.307794 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:57:17.307938 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.303147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.113.2:36717 (local address 127.22.113.1:36443)
0619 06:57:17.303314 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.303319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.303897 (+ 578us) server_negotiation.cc:407] Connection header received
0619 06:57:17.304036 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.304038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.304126 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.304213 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:17.304534 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.305167 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.306562 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.307152 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.307298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.307341 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.307723 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.307794 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:57:17.307991 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.303079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.3:36693 (local address 127.22.113.2:39273)
0619 06:57:17.303336 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.303356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.303472 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.303825 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.303831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.303856 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:17.304018 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.304026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.304991 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.304994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.306328 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.306338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.307338 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.307341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.307536 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.307605 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.307688 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.307808 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":81}
I20260619 06:57:17.307999 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.303092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36443 (local address 127.22.113.2:36717)
0619 06:57:17.303771 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.303787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:17.303917 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:17.304243 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:17.304248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:17.304275 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:17.304402 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.304408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.305270 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.305273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.306434 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:17.306442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.307330 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.307334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:17.307541 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.307603 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 06:57:17.307688 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:57:17.307808 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":358,"spinlock_wait_cycles":10752,"thread_start_us":141,"threads_started":1}
I20260619 06:57:17.308817 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3b52357b4e4f9a937fc2804834db19" candidate_uuid: "5b57713a45c347088d151c953d500764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" is_pre_election: true
I20260619 06:57:17.308837 10393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3b52357b4e4f9a937fc2804834db19" candidate_uuid: "5b57713a45c347088d151c953d500764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0426708f452849fe84e9214e655a0c39" is_pre_election: true
I20260619 06:57:17.309136 10655 raft_consensus.cc:2454] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57713a45c347088d151c953d500764 in term 0.
I20260619 06:57:17.309180 10393 raft_consensus.cc:2454] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b57713a45c347088d151c953d500764 in term 0.
I20260619 06:57:17.309639 10458 leader_election.cc:302] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57713a45c347088d151c953d500764, d4b1c9ecab364b20b1bc05b8946c2be2; no voters:
I20260619 06:57:17.309983 10726 raft_consensus.cc:2781] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:17.310072 10726 raft_consensus.cc:482] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:17.310117 10726 raft_consensus.cc:3037] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:17.311468 10726 raft_consensus.cc:504] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
I20260619 06:57:17.311866 10726 leader_election.cc:288] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [CANDIDATE]: Term 1 election: Requested vote from peers 0426708f452849fe84e9214e655a0c39 (127.22.113.1:36443), d4b1c9ecab364b20b1bc05b8946c2be2 (127.22.113.3:36693)
I20260619 06:57:17.312332 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3b52357b4e4f9a937fc2804834db19" candidate_uuid: "5b57713a45c347088d151c953d500764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2"
I20260619 06:57:17.312355 10393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3b52357b4e4f9a937fc2804834db19" candidate_uuid: "5b57713a45c347088d151c953d500764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0426708f452849fe84e9214e655a0c39"
I20260619 06:57:17.312474 10655 raft_consensus.cc:3037] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:17.312489 10393 raft_consensus.cc:3037] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:17.313659 10393 raft_consensus.cc:2454] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57713a45c347088d151c953d500764 in term 1.
I20260619 06:57:17.313668 10655 raft_consensus.cc:2454] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b57713a45c347088d151c953d500764 in term 1.
I20260619 06:57:17.313990 10458 leader_election.cc:302] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b57713a45c347088d151c953d500764, d4b1c9ecab364b20b1bc05b8946c2be2; no voters:
I20260619 06:57:17.314153 10726 raft_consensus.cc:2781] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:17.314401 10726 raft_consensus.cc:686] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 1 LEADER]: Becoming Leader. State: Replica: 5b57713a45c347088d151c953d500764, State: Running, Role: LEADER
I20260619 06:57:17.314772 10726 consensus_queue.cc:237] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } }
W20260619 06:57:17.318274 10702 tablet.cc:2369] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:17.318549 10246 catalog_manager.cc:5495] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b57713a45c347088d151c953d500764 (127.22.113.2). New cstate: current_term: 1 leader_uuid: "5b57713a45c347088d151c953d500764" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b57713a45c347088d151c953d500764" member_type: VOTER last_known_addr { host: "127.22.113.2" port: 36457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 } health_report { overall_health: UNKNOWN } } }
I20260619 06:57:17.351012 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.347363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.22.113.2:36457)
0619 06:57:17.347512 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:17.347516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:17.347531 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:57:17.347629 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:17.347632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:17.347704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:17.347845 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:17.348074 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.348637 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.349488 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:17.349696 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:17.349777 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:17.350065 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:17.350631 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:17.350731 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:17.350803 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:57:17.350858 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:57:17.358249 10393 raft_consensus.cc:1261] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39 [term 1 FOLLOWER]: Refusing update from remote peer 5b57713a45c347088d151c953d500764: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:17.358250 10655 raft_consensus.cc:1261] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 1 FOLLOWER]: Refusing update from remote peer 5b57713a45c347088d151c953d500764: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:17.359313 10726 consensus_queue.cc:1035] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b1c9ecab364b20b1bc05b8946c2be2" member_type: VOTER last_known_addr { host: "127.22.113.3" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:17.359602 10728 consensus_queue.cc:1035] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0426708f452849fe84e9214e655a0c39" member_type: VOTER last_known_addr { host: "127.22.113.1" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:17.368669 10733 mvcc.cc:205] Tried to move back new op lower bound from 7298466764210012160 to 7298466764044574720. Current Snapshot: MvccSnapshot[applied={T|T < 7298466764210012160}]
I20260619 06:57:17.374404 10731 mvcc.cc:205] Tried to move back new op lower bound from 7298466764210012160 to 7298466764044574720. Current Snapshot: MvccSnapshot[applied={T|T < 7298466764210012160}]
I20260619 06:57:17.383541 10251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:17.383772 10251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:57:17.386838 10251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd42ef09e401440d8cbf20977bb565fc: Sending DeleteTablet for 3 replicas of tablet ed3b52357b4e4f9a937fc2804834db19
I20260619 06:57:17.387950 10504 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3b52357b4e4f9a937fc2804834db19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:17 UTC) from {username='slave'} at 127.0.0.1:50944
I20260619 06:57:17.388113 10373 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3b52357b4e4f9a937fc2804834db19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20260619 06:57:17.388549 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10311
I20260619 06:57:17.388590 10741 tablet_replica.cc:330] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: stopping tablet replica
I20260619 06:57:17.388708 10742 tablet_replica.cc:330] T ed3b52357b4e4f9a937fc2804834db19 P 0426708f452849fe84e9214e655a0c39: stopping tablet replica
I20260619 06:57:17.389065 10741 raft_consensus.cc:2229] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:57:17.389552 10741 raft_consensus.cc:2258] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:17.390972 10741 ts_tablet_manager.cc:1905] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:57:17.393919 10741 ts_tablet_manager.cc:1918] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:57:17.394045 10741 log.cc:1196] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-1/wal/wals/ed3b52357b4e4f9a937fc2804834db19
I20260619 06:57:17.394174 10635 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3b52357b4e4f9a937fc2804834db19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:57:17 UTC) from {username='slave'} at 127.0.0.1:38542
I20260619 06:57:17.394373 10741 ts_tablet_manager.cc:1939] T ed3b52357b4e4f9a937fc2804834db19 P 5b57713a45c347088d151c953d500764: Deleting consensus metadata
I20260619 06:57:17.395694 10743 tablet_replica.cc:330] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: stopping tablet replica
I20260619 06:57:17.395678 10237 catalog_manager.cc:4841] TS 5b57713a45c347088d151c953d500764 (127.22.113.2:36457): tablet ed3b52357b4e4f9a937fc2804834db19 (table test-table [id=cf1e4649c06243e69e976ee1be5d05b6]) successfully deleted
I20260619 06:57:17.396080 10743 raft_consensus.cc:2229] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:57:17.396445 10743 raft_consensus.cc:2258] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:57:17.397933 10743 ts_tablet_manager.cc:1905] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:57:17.399622 10237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.113.1:36443 (error 108)
I20260619 06:57:17.399789 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10442
I20260619 06:57:17.402185 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:17.401737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.1:36443 (local address 127.0.0.1:44260)
0619 06:57:17.401937 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:17.402056 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:57:17.402483 10743 ts_tablet_manager.cc:1918] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 06:57:17.402446 10237 catalog_manager.cc:4584] TS 0426708f452849fe84e9214e655a0c39 (127.22.113.1:36443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed3b52357b4e4f9a937fc2804834db19: Network error: Client connection negotiation failed: client connection to 127.22.113.1:36443: connect: Connection refused (error 111)
I20260619 06:57:17.402602 10743 log.cc:1196] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Deleting WAL directory at /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851976522130-22980-0/minicluster-data/ts-2/wal/wals/ed3b52357b4e4f9a937fc2804834db19
I20260619 06:57:17.402895 10743 ts_tablet_manager.cc:1939] T ed3b52357b4e4f9a937fc2804834db19 P d4b1c9ecab364b20b1bc05b8946c2be2: Deleting consensus metadata
I20260619 06:57:17.404284 10236 catalog_manager.cc:4841] TS d4b1c9ecab364b20b1bc05b8946c2be2 (127.22.113.3:36693): tablet ed3b52357b4e4f9a937fc2804834db19 (table test-table [id=cf1e4649c06243e69e976ee1be5d05b6]) successfully deleted
I20260619 06:57:17.406327 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10573
I20260619 06:57:17.412541 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10220
2026-06-19T06:57:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1276 ms)
[----------] 6 tests from EncryptionPolicyTest (11166 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:17 dist-test-slave-g8zg krb5kdc[10747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:17 dist-test-slave-g8zg krb5kdc[10747](info): set up 2 sockets
Jun 19 06:57:17 dist-test-slave-g8zg krb5kdc[10747](info): commencing operation
krb5kdc: starting...
W20260619 06:57:18.463485 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:18.598634 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:18.719843 10760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:18.720201 10760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:18.720273 10760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:18.724466 10760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:18.724560 10760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:18.724599 10760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:18.724653 10760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:18.724690 10760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:18.730239 10760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:18.731621 10760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:18.732841 10760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:18.742934 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:18.743010 10765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:18.743005 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:18.743218 10760 server_base.cc:1034] running on GCE node
I20260619 06:57:18.744217 10760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:18.745576 10760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:18.746800 10760 hybrid_clock.cc:648] HybridClock initialized: now 1781852238746760 us; error 54 us; skew 500 ppm
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:18.753496 10760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:18.754662 10760 webserver.cc:460] Webserver started at http://127.0.0.1:37645/ using document root <none> and password file <none>
I20260619 06:57:18.755395 10760 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:18.755470 10760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:18.755679 10760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:18.758047 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "94c97ca806534b2f9b925cdd44d665ca"
format_stamp: "Formatted at 2026-06-19 06:57:18 on dist-test-slave-g8zg"
server_key: "dc65b7b2876aef09952aa6d8c6c0b526"
server_key_iv: "992811d2c0e3f842063ceb389dd7e841"
server_key_version: "encryptionkey@0"
I20260619 06:57:18.758764 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "94c97ca806534b2f9b925cdd44d665ca"
format_stamp: "Formatted at 2026-06-19 06:57:18 on dist-test-slave-g8zg"
server_key: "dc65b7b2876aef09952aa6d8c6c0b526"
server_key_iv: "992811d2c0e3f842063ceb389dd7e841"
server_key_version: "encryptionkey@0"
I20260619 06:57:18.763379 10760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:57:18.766192 10775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:18.767838 10760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:18.767985 10760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "94c97ca806534b2f9b925cdd44d665ca"
format_stamp: "Formatted at 2026-06-19 06:57:18 on dist-test-slave-g8zg"
server_key: "dc65b7b2876aef09952aa6d8c6c0b526"
server_key_iv: "992811d2c0e3f842063ceb389dd7e841"
server_key_version: "encryptionkey@0"
I20260619 06:57:18.768107 10760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:18.782773 10760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:18.786602 10760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:18.786839 10760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:18.795634 10760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37949
I20260619 06:57:18.795663 10827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37949 every 8 connection(s)
I20260619 06:57:18.796818 10760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:18.799866 10828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:18.805505 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca: Bootstrap starting.
I20260619 06:57:18.806263 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10760
I20260619 06:57:18.806450 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:18.806780 22980 external_mini_cluster.cc:1428] Setting key f64f9d98ad40c523bf008cf2ecea9f0c
I20260619 06:57:18.808265 10828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:18.809051 10828 log.cc:824] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:18.811499 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca: No bootstrap required, opened a new log
I20260619 06:57:18.815874 10828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } }
I20260619 06:57:18.816205 10828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:18.816259 10828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c97ca806534b2f9b925cdd44d665ca, State: Initialized, Role: FOLLOWER
I20260619 06:57:18.816782 10828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } }
I20260619 06:57:18.816917 10828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:18.817022 10828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:18.817130 10828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:57:18.818150 10828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } }
I20260619 06:57:18.818499 10828 leader_election.cc:302] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c97ca806534b2f9b925cdd44d665ca; no voters:
I20260619 06:57:18.818874 10828 leader_election.cc:288] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:18.818931 10833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:18.819123 10833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [term 1 LEADER]: Becoming Leader. State: Replica: 94c97ca806534b2f9b925cdd44d665ca, State: Running, Role: LEADER
I20260619 06:57:18.819440 10833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } }
I20260619 06:57:18.820202 10828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:18.820621 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c97ca806534b2f9b925cdd44d665ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } } }
I20260619 06:57:18.820736 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:18.821189 10837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:18.821561 10834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c97ca806534b2f9b925cdd44d665ca. Latest consensus state: current_term: 1 leader_uuid: "94c97ca806534b2f9b925cdd44d665ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c97ca806534b2f9b925cdd44d665ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37949 } } }
I20260619 06:57:18.821681 10834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:18.824791 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:18.808250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57199 (local address 127.0.0.1:37949)
0619 06:57:18.808811 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:18.808831 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:18.808877 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:57:18.809969 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:18.810002 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:18.810425 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:18.810767 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:18.811786 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:18.812903 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:18.814150 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:18.814677 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:18.814814 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:18.818139 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:18.818145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:18.818160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:18.818188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:18.820362 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:18.821083 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:18.821094 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:18.821103 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:18.821312 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:18.822084 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:18.822088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:18.822090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:18.822650 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:18.822891 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:18.823471 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 06:57:18.823720 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":172,"threads_started":1}
I20260619 06:57:18.826629 10837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:18.828711 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:18.833655 10837 catalog_manager.cc:1347] Generated new cluster ID: 1af56bb20d984be8a7c28c54e2cf606f
I20260619 06:57:18.833892 10837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:18.840128 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:18.831164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60659 (local address 127.0.0.1:37949)
0619 06:57:18.831322 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:18.831327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:18.831547 (+ 220us) server_negotiation.cc:407] Connection header received
0619 06:57:18.831690 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:18.831693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:18.831742 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:18.831832 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:18.833016 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:18.833830 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:18.834801 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:18.835060 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:18.835176 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:18.836702 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:18.836705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:18.836708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:18.836731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:18.838192 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:18.838672 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:18.838676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:18.838678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:18.838763 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:18.839091 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:18.839094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:18.839096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:18.839346 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:18.839556 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:18.839828 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 06:57:18.839931 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:57:18.845381 10837 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:18.846387 10837 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:18.854563 10837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94c97ca806534b2f9b925cdd44d665ca: Generated new TSK 0
I20260619 06:57:18.855430 10837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:18.860647 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10760
W20260619 06:57:18.866539 10745 connection.cc:535] client connection to 127.0.0.1:37949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37949): Connection reset by peer (error 104)
2026-06-19T06:57:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10859](info): set up 2 sockets
Jun 19 06:57:18 dist-test-slave-g8zg krb5kdc[10859](info): commencing operation
krb5kdc: starting...
W20260619 06:57:19.931533 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:20 dist-test-slave-g8zg krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:20.064761 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:20.195695 10872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:20.196105 10872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:20.196188 10872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:20.200459 10872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:20.200580 10872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:20.200618 10872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:20.200660 10872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:20.200698 10872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:20.206574 10872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:20.208026 10872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:20.209322 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:20.219521 10878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:20.219564 10872 server_base.cc:1034] running on GCE node
W20260619 06:57:20.219506 10877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:20.219506 10880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:20.220599 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:20.221953 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:20.223142 10872 hybrid_clock.cc:648] HybridClock initialized: now 1781852240223107 us; error 52 us; skew 500 ppm
Jun 19 06:57:20 dist-test-slave-g8zg krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:20.226431 10872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:20.227579 10872 webserver.cc:460] Webserver started at http://127.0.0.1:44961/ using document root <none> and password file <none>
I20260619 06:57:20.228268 10872 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:20.228340 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:20.228624 10872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:20.231042 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "0d3d00287e254a199c6688f80de92e5a"
format_stamp: "Formatted at 2026-06-19 06:57:20 on dist-test-slave-g8zg"
server_key: "dfcf7884db8fe9a822413f0f604bd228"
server_key_iv: "b7db063327c5b3ebf8148ec8f10b5bcb"
server_key_version: "encryptionkey@0"
I20260619 06:57:20.231774 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "0d3d00287e254a199c6688f80de92e5a"
format_stamp: "Formatted at 2026-06-19 06:57:20 on dist-test-slave-g8zg"
server_key: "dfcf7884db8fe9a822413f0f604bd228"
server_key_iv: "b7db063327c5b3ebf8148ec8f10b5bcb"
server_key_version: "encryptionkey@0"
I20260619 06:57:20.236313 10872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:20.239385 10887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:20.240957 10872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:57:20.241110 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "0d3d00287e254a199c6688f80de92e5a"
format_stamp: "Formatted at 2026-06-19 06:57:20 on dist-test-slave-g8zg"
server_key: "dfcf7884db8fe9a822413f0f604bd228"
server_key_iv: "b7db063327c5b3ebf8148ec8f10b5bcb"
server_key_version: "encryptionkey@0"
I20260619 06:57:20.241233 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:20.261103 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:20.264976 10872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:20.265210 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:20.275523 10872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40849
I20260619 06:57:20.275652 10939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40849 every 8 connection(s)
I20260619 06:57:20.276762 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:20.280746 10940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:20.282634 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10872
I20260619 06:57:20.282850 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:20.283185 22980 external_mini_cluster.cc:1428] Setting key f5e552aef1a5c382086b15254a61f802
I20260619 06:57:20.288908 10940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: Bootstrap starting.
I20260619 06:57:20.292119 10940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:20.293220 10940 log.cc:824] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:20.293457 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:20.284749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54149 (local address 127.0.0.1:40849)
0619 06:57:20.285271 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:20.285322 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:20.285366 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:57:20.286139 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:20.286158 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:20.286578 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:20.286953 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:20.287467 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:20.288518 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:20.289516 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:20.290032 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:20.290212 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:20.291205 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:20.291210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:20.291224 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:20.291752 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:20.291815 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:20.291839 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:20.291985 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:20.292156 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 06:57:20.292415 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":180,"threads_started":1}
I20260619 06:57:20.295912 10940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: No bootstrap required, opened a new log
I20260619 06:57:20.300788 10940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260619 06:57:20.301090 10940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:20.301136 10940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3d00287e254a199c6688f80de92e5a, State: Initialized, Role: FOLLOWER
I20260619 06:57:20.301676 10940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260619 06:57:20.301828 10940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:20.301882 10940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:20.301971 10940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:20.303419 10940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260619 06:57:20.303846 10940 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3d00287e254a199c6688f80de92e5a; no voters:
I20260619 06:57:20.304329 10940 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:20.304409 10945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:20.304625 10945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [term 1 LEADER]: Becoming Leader. State: Replica: 0d3d00287e254a199c6688f80de92e5a, State: Running, Role: LEADER
I20260619 06:57:20.305356 10945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260619 06:57:20.305872 10940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:20.306842 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:20.309815 10947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3d00287e254a199c6688f80de92e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260619 06:57:20.310038 10947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:20.310900 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3d00287e254a199c6688f80de92e5a. Latest consensus state: current_term: 1 leader_uuid: "0d3d00287e254a199c6688f80de92e5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3d00287e254a199c6688f80de92e5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260619 06:57:20.311045 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a [sys.catalog]: This master's current role is: LEADER
W20260619 06:57:20.312078 10959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:20.312259 10959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:20.312381 10965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:20.313452 10965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:20.317857 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:20.311698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39521 (local address 127.0.0.1:40849)
0619 06:57:20.311819 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:20.311824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:20.312396 (+ 572us) server_negotiation.cc:407] Connection header received
0619 06:57:20.312622 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:20.312626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:20.312680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:20.312737 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:20.313283 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:20.314152 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:20.315472 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:20.315882 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:20.315970 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:20.317219 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:20.317224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:20.317228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:20.317242 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:20.317274 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:20.317289 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:20.317355 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:20.317610 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 06:57:20.317657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:57:20.320740 10965 catalog_manager.cc:1347] Generated new cluster ID: 2046b06e477145f39713a150820f61ad
I20260619 06:57:20.320871 10965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:20.345394 10965 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:20.347024 10965 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:20.357740 10965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3d00287e254a199c6688f80de92e5a: Generated new TSK 0
I20260619 06:57:20.358664 10965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:20.363281 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10872
W20260619 06:57:20.368984 10857 connection.cc:535] client connection to 127.0.0.1:40849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40849): Connection reset by peer (error 104)
2026-06-19T06:57:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1502 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:20 dist-test-slave-g8zg krb5kdc[10971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:20 dist-test-slave-g8zg krb5kdc[10971](info): set up 2 sockets
Jun 19 06:57:20 dist-test-slave-g8zg krb5kdc[10971](info): commencing operation
krb5kdc: starting...
W20260619 06:57:21.411841 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:21 dist-test-slave-g8zg krb5kdc[10971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:21.546221 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:21.665177 10984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:21.665537 10984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:21.665620 10984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:21.671217 10984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:21.671453 10984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:21.671499 10984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:21.671545 10984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:21.671587 10984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:21.678220 10984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:21.679723 10984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:21.680974 10984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:21.691118 10990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:21.691116 10989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:21.691284 10984 server_base.cc:1034] running on GCE node
W20260619 06:57:21.691176 10992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:21.692252 10984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:21.693722 10984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:21.694952 10984 hybrid_clock.cc:648] HybridClock initialized: now 1781852241694918 us; error 48 us; skew 500 ppm
Jun 19 06:57:21 dist-test-slave-g8zg krb5kdc[10971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:21.703802 10984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:21.705085 10984 webserver.cc:460] Webserver started at http://127.0.0.1:46593/ using document root <none> and password file <none>
I20260619 06:57:21.705804 10984 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:21.705909 10984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:21.706187 10984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:21.708776 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "325eeb2d4bc74a01b29eb64a3d0b3169"
format_stamp: "Formatted at 2026-06-19 06:57:21 on dist-test-slave-g8zg"
server_key: "f3e9c6eb91e6eebeae9efa776bab4614"
server_key_iv: "097dc3b0a44a9b0a26d777d9a3737ca2"
server_key_version: "encryptionkey@0"
I20260619 06:57:21.709419 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "325eeb2d4bc74a01b29eb64a3d0b3169"
format_stamp: "Formatted at 2026-06-19 06:57:21 on dist-test-slave-g8zg"
server_key: "f3e9c6eb91e6eebeae9efa776bab4614"
server_key_iv: "097dc3b0a44a9b0a26d777d9a3737ca2"
server_key_version: "encryptionkey@0"
I20260619 06:57:21.713971 10984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:21.716997 10999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:21.718616 10984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:21.718782 10984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "325eeb2d4bc74a01b29eb64a3d0b3169"
format_stamp: "Formatted at 2026-06-19 06:57:21 on dist-test-slave-g8zg"
server_key: "f3e9c6eb91e6eebeae9efa776bab4614"
server_key_iv: "097dc3b0a44a9b0a26d777d9a3737ca2"
server_key_version: "encryptionkey@0"
I20260619 06:57:21.718921 10984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:21.724368 10984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:21.728091 10984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:21.728309 10984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:21.736912 10984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34375
I20260619 06:57:21.736943 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34375 every 8 connection(s)
I20260619 06:57:21.738001 10984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:21.740896 11052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:21.744212 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 10984
I20260619 06:57:21.744405 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:21.744801 22980 external_mini_cluster.cc:1428] Setting key d9c3ecc1bbccc49484b4d05d41816c3e
I20260619 06:57:21.747071 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: Bootstrap starting.
I20260619 06:57:21.749644 11052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:21.750666 11052 log.cc:824] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:21.751953 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:21.746671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35895 (local address 127.0.0.1:34375)
0619 06:57:21.747226 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:21.747251 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:21.747326 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:57:21.748125 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:21.748145 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:21.748491 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:21.748835 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:21.749924 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:21.749930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:21.749942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:21.749990 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:21.750086 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:21.750119 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:21.750269 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:21.750492 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:57:21.750706 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":168,"threads_started":1}
I20260619 06:57:21.753552 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: No bootstrap required, opened a new log
I20260619 06:57:21.756422 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:21.758729 11052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260619 06:57:21.759029 11052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:21.759097 11052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325eeb2d4bc74a01b29eb64a3d0b3169, State: Initialized, Role: FOLLOWER
I20260619 06:57:21.759598 11052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260619 06:57:21.759784 11052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:21.759859 11052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:21.759970 11052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:21.761117 11052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260619 06:57:21.761505 11052 leader_election.cc:302] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325eeb2d4bc74a01b29eb64a3d0b3169; no voters:
I20260619 06:57:21.761897 11052 leader_election.cc:288] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:21.762077 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:21.759373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40615 (local address 127.0.0.1:34375)
0619 06:57:21.759552 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:21.759557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:21.760079 (+ 522us) server_negotiation.cc:407] Connection header received
0619 06:57:21.760298 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:21.760303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:21.760352 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:21.760408 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:21.761445 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:21.761450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:21.761454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:21.761474 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:21.761520 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:21.761540 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:21.761677 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:21.761886 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 06:57:21.761944 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 06:57:21.762091 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:21.762466 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [term 1 LEADER]: Becoming Leader. State: Replica: 325eeb2d4bc74a01b29eb64a3d0b3169, State: Running, Role: LEADER
I20260619 06:57:21.762905 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260619 06:57:21.763229 11052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:21.765630 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325eeb2d4bc74a01b29eb64a3d0b3169. Latest consensus state: current_term: 1 leader_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } } }
I20260619 06:57:21.765743 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325eeb2d4bc74a01b29eb64a3d0b3169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } } }
I20260619 06:57:21.766603 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:21.766623 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169 [sys.catalog]: This master's current role is: LEADER
W20260619 06:57:21.768494 11076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:21.768586 11076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:21.768692 11078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:21.769802 11078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:21.776235 11078 catalog_manager.cc:1347] Generated new cluster ID: 3ca555656bf0410fa152a3a7e81d25e1
I20260619 06:57:21.776329 11078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:21.786494 11078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:21.786875 11078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:21.792645 11078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325eeb2d4bc74a01b29eb64a3d0b3169: Generated new TSK 0
I20260619 06:57:21.793548 11078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:21.797528 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 10984
2026-06-19T06:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:21 dist-test-slave-g8zg krb5kdc[11083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:21 dist-test-slave-g8zg krb5kdc[11083](info): set up 2 sockets
Jun 19 06:57:21 dist-test-slave-g8zg krb5kdc[11083](info): commencing operation
krb5kdc: starting...
W20260619 06:57:22.851488 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:22 dist-test-slave-g8zg krb5kdc[11083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:22.981945 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:23.104657 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:23.104962 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:23.105007 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:23.109023 11096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:23.109088 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:23.109108 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:23.109127 11096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:23.109143 11096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:23.114540 11096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:23.115768 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:23.116958 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:23.127497 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:23.127560 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:23.127497 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:23.127975 11096 server_base.cc:1034] running on GCE node
I20260619 06:57:23.128700 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:23.129932 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:23.131134 11096 hybrid_clock.cc:648] HybridClock initialized: now 1781852243131114 us; error 42 us; skew 500 ppm
Jun 19 06:57:23 dist-test-slave-g8zg krb5kdc[11083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:23.134377 11096 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:23.135418 11096 webserver.cc:460] Webserver started at http://127.0.0.1:46197/ using document root <none> and password file <none>
I20260619 06:57:23.136060 11096 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:23.136152 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:23.136394 11096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:23.138489 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "86f0f8b0b930442ca02f0284ebc78bb3"
format_stamp: "Formatted at 2026-06-19 06:57:23 on dist-test-slave-g8zg"
server_key: "554e6203b7e91e376b245f5aa3a381b6"
server_key_iv: "ddd71384375dfc2a4a5ae7b5f1593ec9"
server_key_version: "encryptionkey@0"
I20260619 06:57:23.139079 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "86f0f8b0b930442ca02f0284ebc78bb3"
format_stamp: "Formatted at 2026-06-19 06:57:23 on dist-test-slave-g8zg"
server_key: "554e6203b7e91e376b245f5aa3a381b6"
server_key_iv: "ddd71384375dfc2a4a5ae7b5f1593ec9"
server_key_version: "encryptionkey@0"
I20260619 06:57:23.143098 11096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 06:57:23.145780 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:23.147379 11096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:23.147522 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "86f0f8b0b930442ca02f0284ebc78bb3"
format_stamp: "Formatted at 2026-06-19 06:57:23 on dist-test-slave-g8zg"
server_key: "554e6203b7e91e376b245f5aa3a381b6"
server_key_iv: "ddd71384375dfc2a4a5ae7b5f1593ec9"
server_key_version: "encryptionkey@0"
I20260619 06:57:23.147644 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:23.169064 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:23.172546 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:23.172752 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:23.180620 11096 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41621
I20260619 06:57:23.180760 11163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41621 every 8 connection(s)
I20260619 06:57:23.181656 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:23.185091 11164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:23.189635 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11096
I20260619 06:57:23.189805 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:23.190179 22980 external_mini_cluster.cc:1428] Setting key 7f6448299dc3341d410e75708989ab9c
I20260619 06:57:23.192198 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3: Bootstrap starting.
I20260619 06:57:23.194902 11164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:23.195878 11164 log.cc:824] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:23.198251 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3: No bootstrap required, opened a new log
Jun 19 06:57:23 dist-test-slave-g8zg krb5kdc[11083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:57:23.202843 11164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260619 06:57:23.203099 11164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:23.203195 11164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f0f8b0b930442ca02f0284ebc78bb3, State: Initialized, Role: FOLLOWER
I20260619 06:57:23.203795 11164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260619 06:57:23.204008 11164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:23.204108 11164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:23.204237 11164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:23.205610 11164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260619 06:57:23.206032 11164 leader_election.cc:302] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f0f8b0b930442ca02f0284ebc78bb3; no voters:
I20260619 06:57:23.206405 11164 leader_election.cc:288] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:23.206552 11169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:23.206552 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:23.191918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42231 (local address 127.0.0.1:41621)
0619 06:57:23.192548 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:23.192567 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:23.192608 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:57:23.193464 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:23.193484 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:23.193875 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:23.194254 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:23.195104 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:23.196275 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:23.197340 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:23.197825 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:23.198010 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:23.201090 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:23.201094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:23.201109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:23.201136 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:23.203034 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:23.203653 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:23.203658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:23.203662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:23.203773 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:23.204145 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:23.204148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:23.204149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:23.204644 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:23.204807 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:23.205173 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:57:23.205565 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":167,"threads_started":1}
I20260619 06:57:23.206859 11169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 86f0f8b0b930442ca02f0284ebc78bb3, State: Running, Role: LEADER
I20260619 06:57:23.207273 11169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260619 06:57:23.207976 11164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:23.209028 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } } }
I20260619 06:57:23.209180 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:23.209445 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f0f8b0b930442ca02f0284ebc78bb3. Latest consensus state: current_term: 1 leader_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f0f8b0b930442ca02f0284ebc78bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } } }
I20260619 06:57:23.209543 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:23.209833 11178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:23.209955 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:23.213397 11178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:23.220155 11178 catalog_manager.cc:1347] Generated new cluster ID: 162f62850e9e4076859a127555a50b85
I20260619 06:57:23.220229 11178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:23.222084 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:23.212157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56003 (local address 127.0.0.1:41621)
0619 06:57:23.212289 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:23.212294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:23.212583 (+ 289us) server_negotiation.cc:407] Connection header received
0619 06:57:23.212712 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:23.212716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:23.212776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:23.212887 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:23.213774 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:23.214548 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:23.215711 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:23.216130 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:23.216256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:23.218373 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:23.218377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:23.218379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:23.218411 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:23.220288 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:23.220775 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:23.220779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:23.220781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:23.220868 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:23.221158 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:23.221162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:23.221163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:23.221371 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:23.221514 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:23.221748 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:57:23.221880 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":46}
I20260619 06:57:23.238933 11178 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:23.240078 11178 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:23.244671 11178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f0f8b0b930442ca02f0284ebc78bb3: Generated new TSK 0
I20260619 06:57:23.245329 11178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:23.251756 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11096
2026-06-19T06:57:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:23 dist-test-slave-g8zg krb5kdc[11195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:23 dist-test-slave-g8zg krb5kdc[11195](info): set up 2 sockets
Jun 19 06:57:23 dist-test-slave-g8zg krb5kdc[11195](info): commencing operation
krb5kdc: starting...
W20260619 06:57:24.303771 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:24 dist-test-slave-g8zg krb5kdc[11195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:24.436506 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:24.550897 11208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:24.551198 11208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:24.551298 11208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:24.555301 11208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:24.555361 11208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:24.555382 11208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:24.555398 11208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:24.555414 11208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:24.560567 11208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:24.561523 11208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:24.562479 11208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:24.572196 11214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:24.572206 11216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:24.572233 11213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:24.572395 11208 server_base.cc:1034] running on GCE node
I20260619 06:57:24.573164 11208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:24.574208 11208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:24.575400 11208 hybrid_clock.cc:648] HybridClock initialized: now 1781852244575373 us; error 54 us; skew 500 ppm
Jun 19 06:57:24 dist-test-slave-g8zg krb5kdc[11195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:24.578238 11208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:24.579113 11208 webserver.cc:460] Webserver started at http://127.0.0.1:42787/ using document root <none> and password file <none>
I20260619 06:57:24.579691 11208 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:24.579756 11208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:24.579914 11208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:24.581796 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "636bc0be8b0541348ba6f97c01529eb3"
format_stamp: "Formatted at 2026-06-19 06:57:24 on dist-test-slave-g8zg"
server_key: "aa5e612d40cfdde8d832c2885b58fac1"
server_key_iv: "8b6495e01acc13b43642f1e14554c93d"
server_key_version: "encryptionkey@0"
I20260619 06:57:24.582288 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "636bc0be8b0541348ba6f97c01529eb3"
format_stamp: "Formatted at 2026-06-19 06:57:24 on dist-test-slave-g8zg"
server_key: "aa5e612d40cfdde8d832c2885b58fac1"
server_key_iv: "8b6495e01acc13b43642f1e14554c93d"
server_key_version: "encryptionkey@0"
I20260619 06:57:24.585949 11208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:57:24.588362 11223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:24.589638 11208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:24.589751 11208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "636bc0be8b0541348ba6f97c01529eb3"
format_stamp: "Formatted at 2026-06-19 06:57:24 on dist-test-slave-g8zg"
server_key: "aa5e612d40cfdde8d832c2885b58fac1"
server_key_iv: "8b6495e01acc13b43642f1e14554c93d"
server_key_version: "encryptionkey@0"
I20260619 06:57:24.589833 11208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:24.613569 11208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:24.616551 11208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:24.616706 11208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:24.624145 11208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33669
I20260619 06:57:24.624238 11275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33669 every 8 connection(s)
I20260619 06:57:24.625118 11208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:24.628558 11276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:24.634107 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11208
I20260619 06:57:24.634238 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:24.634608 22980 external_mini_cluster.cc:1428] Setting key 80744b076ae5f7c2f218e8a27172d0eb
I20260619 06:57:24.635202 11276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3: Bootstrap starting.
I20260619 06:57:24.637935 11276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:24.638899 11276 log.cc:824] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:24.641252 11276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3: No bootstrap required, opened a new log
I20260619 06:57:24.644317 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:24.636076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36783 (local address 127.0.0.1:33669)
0619 06:57:24.636545 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:24.636561 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:24.636600 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:57:24.637374 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:24.637400 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:24.637760 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:24.638088 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:24.638465 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:24.639564 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:24.640542 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:24.641185 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:24.641356 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:24.642281 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:24.642287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:24.642302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:24.642794 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:24.642858 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:24.642895 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:24.642997 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:24.643121 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 06:57:24.643354 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":146,"threads_started":1}
I20260619 06:57:24.645902 11276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260619 06:57:24.646123 11276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:24.646193 11276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636bc0be8b0541348ba6f97c01529eb3, State: Initialized, Role: FOLLOWER
I20260619 06:57:24.646703 11276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260619 06:57:24.646898 11276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:24.646984 11276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:24.647087 11276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:24.647773 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:24.648296 11276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260619 06:57:24.648639 11276 leader_election.cc:302] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 636bc0be8b0541348ba6f97c01529eb3; no voters:
I20260619 06:57:24.648962 11276 leader_election.cc:288] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:24.649111 11281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:24.649385 11281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 636bc0be8b0541348ba6f97c01529eb3, State: Running, Role: LEADER
I20260619 06:57:24.649773 11281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260619 06:57:24.650440 11276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:24.651340 11287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 636bc0be8b0541348ba6f97c01529eb3. Latest consensus state: current_term: 1 leader_uuid: "636bc0be8b0541348ba6f97c01529eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260619 06:57:24.651563 11287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:24.652268 11285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "636bc0be8b0541348ba6f97c01529eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636bc0be8b0541348ba6f97c01529eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260619 06:57:24.652446 11285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:24.653995 11295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:24.655576 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:24.650235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44413 (local address 127.0.0.1:33669)
0619 06:57:24.650338 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:24.650343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:24.650785 (+ 442us) server_negotiation.cc:407] Connection header received
0619 06:57:24.650909 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:24.650912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:24.650960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:24.651065 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:24.651599 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:24.652395 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:24.653614 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:24.653984 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:24.654117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:24.655043 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:24.655048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:24.655052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:24.655069 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:24.655102 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:24.655115 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:24.655191 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:24.655400 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 06:57:24.655453 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:57:24.657153 11295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:24.662853 11295 catalog_manager.cc:1347] Generated new cluster ID: 8f7e895f528440ac810f6cb3895479bb
I20260619 06:57:24.662919 11295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:24.681136 11295 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:24.682641 11295 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:24.699188 11295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 636bc0be8b0541348ba6f97c01529eb3: Generated new TSK 0
I20260619 06:57:24.700194 11295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:24.712137 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11208
2026-06-19T06:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:24 dist-test-slave-g8zg krb5kdc[11307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:24 dist-test-slave-g8zg krb5kdc[11307](info): set up 2 sockets
Jun 19 06:57:24 dist-test-slave-g8zg krb5kdc[11307](info): commencing operation
krb5kdc: starting...
W20260619 06:57:25.767717 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:25 dist-test-slave-g8zg krb5kdc[11307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:25.900274 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:26.023696 11320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:26.024084 11320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:26.024176 11320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:26.028322 11320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:26.028424 11320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:26.028463 11320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:26.028527 11320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:26.028566 11320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:26.034464 11320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:26.035918 11320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:26.037211 11320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:26.047084 11325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:26.047093 11326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:26.047333 11328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:26.047395 11320 server_base.cc:1034] running on GCE node
I20260619 06:57:26.048252 11320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:26.049516 11320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:26.050725 11320 hybrid_clock.cc:648] HybridClock initialized: now 1781852246050691 us; error 55 us; skew 500 ppm
Jun 19 06:57:26 dist-test-slave-g8zg krb5kdc[11307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:26.053748 11320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:26.054636 11320 webserver.cc:460] Webserver started at http://127.0.0.1:41715/ using document root <none> and password file <none>
I20260619 06:57:26.055178 11320 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:26.055296 11320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:26.055600 11320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:26.057711 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "91b14b9567bf409e8a2b6620e2c5d989"
format_stamp: "Formatted at 2026-06-19 06:57:26 on dist-test-slave-g8zg"
server_key: "78c3bea3382e05b51199b9d899362362"
server_key_iv: "71c777e34833b440c404b5349221bec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:26.058284 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "91b14b9567bf409e8a2b6620e2c5d989"
format_stamp: "Formatted at 2026-06-19 06:57:26 on dist-test-slave-g8zg"
server_key: "78c3bea3382e05b51199b9d899362362"
server_key_iv: "71c777e34833b440c404b5349221bec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:26.062224 11320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 06:57:26.064988 11335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:26.066449 11320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:26.066599 11320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "91b14b9567bf409e8a2b6620e2c5d989"
format_stamp: "Formatted at 2026-06-19 06:57:26 on dist-test-slave-g8zg"
server_key: "78c3bea3382e05b51199b9d899362362"
server_key_iv: "71c777e34833b440c404b5349221bec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:26.066737 11320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:26.071379 11320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:26.074589 11320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:26.074790 11320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:26.082466 11320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38721
I20260619 06:57:26.082494 11387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38721 every 8 connection(s)
I20260619 06:57:26.083485 11320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:26.086831 11388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:26.087700 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11320
I20260619 06:57:26.087877 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:26.088241 22980 external_mini_cluster.cc:1428] Setting key 52e9948912042f9f3bb393f2b31c0948
I20260619 06:57:26.094161 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:26.089877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35455 (local address 127.0.0.1:38721)
0619 06:57:26.090433 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:26.090449 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:26.090488 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:57:26.091363 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:26.091383 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:26.091697 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:26.092063 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:26.092886 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:26.092891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:26.092902 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:26.092912 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:26.092965 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:26.092984 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:26.093178 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:26.093274 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 06:57:26.093433 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":171,"threads_started":1}
I20260619 06:57:26.094348 11388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989: Bootstrap starting.
I20260619 06:57:26.097368 11388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:26.097460 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:26.098212 11388 log.cc:824] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:26.100706 11388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989: No bootstrap required, opened a new log
I20260619 06:57:26.101580 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:26.099719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42161 (local address 127.0.0.1:38721)
0619 06:57:26.099848 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:26.099859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:26.100041 (+ 182us) server_negotiation.cc:407] Connection header received
0619 06:57:26.100182 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:26.100184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:26.100217 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:26.100293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:26.101142 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:26.101146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:26.101148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:26.101159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:26.101189 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:26.101199 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:26.101286 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:26.101398 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:57:26.101436 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:57:26.104885 11388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260619 06:57:26.105132 11388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:26.105196 11388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b14b9567bf409e8a2b6620e2c5d989, State: Initialized, Role: FOLLOWER
I20260619 06:57:26.105683 11388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260619 06:57:26.105818 11388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:26.105908 11388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:26.106043 11388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:26.107031 11388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260619 06:57:26.107465 11388 leader_election.cc:302] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91b14b9567bf409e8a2b6620e2c5d989; no voters:
I20260619 06:57:26.107851 11388 leader_election.cc:288] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:26.108035 11398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:26.108421 11398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [term 1 LEADER]: Becoming Leader. State: Replica: 91b14b9567bf409e8a2b6620e2c5d989, State: Running, Role: LEADER
I20260619 06:57:26.108918 11398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260619 06:57:26.109357 11388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:26.110716 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91b14b9567bf409e8a2b6620e2c5d989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260619 06:57:26.110857 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:26.111014 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b14b9567bf409e8a2b6620e2c5d989. Latest consensus state: current_term: 1 leader_uuid: "91b14b9567bf409e8a2b6620e2c5d989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b14b9567bf409e8a2b6620e2c5d989" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260619 06:57:26.111102 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:26.111425 11408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:26.114523 11408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:26.120818 11408 catalog_manager.cc:1347] Generated new cluster ID: 321826c6277a4e6697210bf78f380593
I20260619 06:57:26.120894 11408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:26.128648 11408 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:26.129336 11408 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:26.140398 11408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91b14b9567bf409e8a2b6620e2c5d989: Generated new TSK 0
I20260619 06:57:26.141319 11408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:26.148113 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11320
2026-06-19T06:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:26 dist-test-slave-g8zg krb5kdc[11419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:26 dist-test-slave-g8zg krb5kdc[11419](info): set up 2 sockets
Jun 19 06:57:26 dist-test-slave-g8zg krb5kdc[11419](info): commencing operation
krb5kdc: starting...
W20260619 06:57:27.200246 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:27.328868 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:27.442852 11432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:27.443133 11432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:27.443178 11432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:27.447285 11432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:27.447368 11432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:27.447389 11432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:27.447407 11432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:27.447466 11432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:27.452683 11432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:27.453804 11432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:27.454929 11432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:27.464522 11438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:27.464576 11437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:27.464618 11432 server_base.cc:1034] running on GCE node
W20260619 06:57:27.464520 11440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:27.465646 11432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:27.467027 11432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:27.468236 11432 hybrid_clock.cc:648] HybridClock initialized: now 1781852247468218 us; error 47 us; skew 500 ppm
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:27.475060 11432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:27.476634 11432 webserver.cc:460] Webserver started at http://127.0.0.1:43331/ using document root <none> and password file <none>
I20260619 06:57:27.477470 11432 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:27.477541 11432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:27.477748 11432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:27.480254 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "4fc973aa600944f3b2f4a413f7050c79"
format_stamp: "Formatted at 2026-06-19 06:57:27 on dist-test-slave-g8zg"
server_key: "3f06563185ba106c0afcb6688328418b"
server_key_iv: "20d645f17ad73c26b273acbad2ce59e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:27.480839 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "4fc973aa600944f3b2f4a413f7050c79"
format_stamp: "Formatted at 2026-06-19 06:57:27 on dist-test-slave-g8zg"
server_key: "3f06563185ba106c0afcb6688328418b"
server_key_iv: "20d645f17ad73c26b273acbad2ce59e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:27.485360 11432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:27.488430 11447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:27.489984 11432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:57:27.490149 11432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "4fc973aa600944f3b2f4a413f7050c79"
format_stamp: "Formatted at 2026-06-19 06:57:27 on dist-test-slave-g8zg"
server_key: "3f06563185ba106c0afcb6688328418b"
server_key_iv: "20d645f17ad73c26b273acbad2ce59e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:27.490280 11432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:27.501443 11432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:27.505682 11432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:27.505942 11432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:27.514745 11432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37487
I20260619 06:57:27.514770 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37487 every 8 connection(s)
I20260619 06:57:27.516029 11432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:27.519043 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:27.524636 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: Bootstrap starting.
I20260619 06:57:27.525890 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11432
I20260619 06:57:27.526054 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:27.526364 22980 external_mini_cluster.cc:1428] Setting key 152c7c1baf903a4620d69c42a9026ba1
I20260619 06:57:27.527521 11500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:27.528462 11500 log.cc:824] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:27.530767 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: No bootstrap required, opened a new log
I20260619 06:57:27.534896 11500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260619 06:57:27.535111 11500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:27.535146 11500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc973aa600944f3b2f4a413f7050c79, State: Initialized, Role: FOLLOWER
I20260619 06:57:27.535611 11500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260619 06:57:27.535732 11500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:27.535811 11500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:27.535933 11500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:57:27.537151 11500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260619 06:57:27.537535 11500 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc973aa600944f3b2f4a413f7050c79; no voters:
I20260619 06:57:27.537880 11500 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:27.538053 11505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:27.538710 11505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc973aa600944f3b2f4a413f7050c79, State: Running, Role: LEADER
I20260619 06:57:27.539146 11500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:27.539494 11505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } }
I20260619 06:57:27.541953 11506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc973aa600944f3b2f4a413f7050c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260619 06:57:27.542109 11506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:27.542440 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc973aa600944f3b2f4a413f7050c79. Latest consensus state: current_term: 1 leader_uuid: "4fc973aa600944f3b2f4a413f7050c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc973aa600944f3b2f4a413f7050c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37487 } } }
I20260619 06:57:27.542541 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:27.544616 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:27.527753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:59341 (local address 127.0.0.1:37487)
0619 06:57:27.528377 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:27.528396 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:27.528441 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:57:27.529231 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:27.529250 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:27.529683 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:27.530073 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:27.530836 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:27.532340 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:27.533279 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:27.534073 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:27.534248 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:27.537813 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:27.537821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:27.537843 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:27.537910 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:27.540325 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:27.541092 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:27.541099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:27.541106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:27.541275 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:27.541951 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:27.541954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:27.541956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:27.542535 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:27.542782 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:27.543296 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 06:57:27.543568 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":157,"threads_started":1}
W20260619 06:57:27.545003 11519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:27.545120 11519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:27.545200 11520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:27.546149 11520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:27.548442 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:27.553433 11520 catalog_manager.cc:1347] Generated new cluster ID: bbd6357da8624404a6bb052c573f5eec
I20260619 06:57:27.553505 11520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:27.561487 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:27.550859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:34421 (local address 127.0.0.1:37487)
0619 06:57:27.550975 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:27.550979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:27.551484 (+ 505us) server_negotiation.cc:407] Connection header received
0619 06:57:27.551637 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:27.551640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:27.551697 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:27.551796 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:27.552703 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:27.553589 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:27.554865 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:27.555223 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:27.555336 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:27.557368 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:27.557372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:27.557376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:27.557406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:27.559289 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:27.560038 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:27.560042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:27.560044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:27.560138 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:27.560533 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:27.560537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:27.560538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:27.560755 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:27.560905 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:27.561209 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 06:57:27.561317 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:57:27.576037 11520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:27.577442 11520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:27.582917 11520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc973aa600944f3b2f4a413f7050c79: Generated new TSK 0
I20260619 06:57:27.583925 11520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:27.588953 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11432
W20260619 06:57:27.594049 11417 connection.cc:535] client connection to 127.0.0.1:37487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37487): Connection reset by peer (error 104)
2026-06-19T06:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11531](info): set up 2 sockets
Jun 19 06:57:27 dist-test-slave-g8zg krb5kdc[11531](info): commencing operation
krb5kdc: starting...
W20260619 06:57:28.639760 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:28 dist-test-slave-g8zg krb5kdc[11531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:28.774890 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:28.884166 11544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:28.884438 11544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:28.884481 11544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:28.888566 11544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:28.888628 11544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:28.888648 11544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:28.888664 11544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:28.888680 11544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:28.893777 11544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:28.894722 11544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:28.895668 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:28.904409 11549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:28.904374 11550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:28.904387 11552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:28.904784 11544 server_base.cc:1034] running on GCE node
I20260619 06:57:28.905355 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:28.906472 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:28.907666 11544 hybrid_clock.cc:648] HybridClock initialized: now 1781852248907643 us; error 42 us; skew 500 ppm
Jun 19 06:57:28 dist-test-slave-g8zg krb5kdc[11531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:28.910758 11544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:28.911783 11544 webserver.cc:460] Webserver started at http://127.0.0.1:38427/ using document root <none> and password file <none>
I20260619 06:57:28.912397 11544 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:28.912492 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:28.912725 11544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:28.914734 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "87bda2e1125a485a9fd9fb1cb9aa220b"
format_stamp: "Formatted at 2026-06-19 06:57:28 on dist-test-slave-g8zg"
server_key: "a5485022ae48c11295bf1c17b333fa5f"
server_key_iv: "d0548856818ac2a28af79a9965bc1bd8"
server_key_version: "encryptionkey@0"
I20260619 06:57:28.915372 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "87bda2e1125a485a9fd9fb1cb9aa220b"
format_stamp: "Formatted at 2026-06-19 06:57:28 on dist-test-slave-g8zg"
server_key: "a5485022ae48c11295bf1c17b333fa5f"
server_key_iv: "d0548856818ac2a28af79a9965bc1bd8"
server_key_version: "encryptionkey@0"
I20260619 06:57:28.919306 11544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:57:28.921869 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:28.923229 11544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:28.923432 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "87bda2e1125a485a9fd9fb1cb9aa220b"
format_stamp: "Formatted at 2026-06-19 06:57:28 on dist-test-slave-g8zg"
server_key: "a5485022ae48c11295bf1c17b333fa5f"
server_key_iv: "d0548856818ac2a28af79a9965bc1bd8"
server_key_version: "encryptionkey@0"
I20260619 06:57:28.923539 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:28.944133 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:28.947175 11544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:28.947403 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:28.955026 11544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33657
I20260619 06:57:28.955054 11611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33657 every 8 connection(s)
I20260619 06:57:28.956044 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:28.959316 11612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:28.962340 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11544
I20260619 06:57:28.962497 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:28.962812 22980 external_mini_cluster.cc:1428] Setting key 8f627a088462eb38bf95363d9919d075
I20260619 06:57:28.966347 11612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b: Bootstrap starting.
I20260619 06:57:28.969023 11612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:28.969893 11612 log.cc:824] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:28.972232 11612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b: No bootstrap required, opened a new log
I20260619 06:57:28.972573 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:28.964413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:54399 (local address 127.0.0.1:33657)
0619 06:57:28.964962 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:28.964980 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:28.965020 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:57:28.965848 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:28.965868 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:28.966266 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:28.966641 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:28.967078 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:28.968143 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:28.969029 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:28.969596 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:28.969774 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:28.970698 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:28.970703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:28.970718 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:28.971138 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:28.971196 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:28.971219 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:28.971428 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:28.971605 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:57:28.971758 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":184,"threads_started":1}
I20260619 06:57:28.976174 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:28.976820 11612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260619 06:57:28.977110 11612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:28.977205 11612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bda2e1125a485a9fd9fb1cb9aa220b, State: Initialized, Role: FOLLOWER
I20260619 06:57:28.977723 11612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260619 06:57:28.977913 11612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:28.978001 11612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:28.978113 11612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:28.979363 11612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260619 06:57:28.979734 11612 leader_election.cc:302] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87bda2e1125a485a9fd9fb1cb9aa220b; no voters:
I20260619 06:57:28.980110 11612 leader_election.cc:288] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:28.980182 11622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:28.980412 11622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [term 1 LEADER]: Becoming Leader. State: Replica: 87bda2e1125a485a9fd9fb1cb9aa220b, State: Running, Role: LEADER
I20260619 06:57:28.980790 11622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } }
I20260619 06:57:28.981582 11612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:28.982815 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:28.978372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:47825 (local address 127.0.0.1:33657)
0619 06:57:28.978516 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:28.978522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:28.978616 (+ 94us) server_negotiation.cc:407] Connection header received
0619 06:57:28.978736 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:28.978739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:28.978799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:28.978882 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:28.979164 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:28.979881 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:28.980944 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:28.981207 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:28.981317 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:28.982280 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:28.982284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:28.982286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:28.982300 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:28.982324 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:28.982336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:28.982397 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:28.982636 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 06:57:28.982672 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260619 06:57:28.982841 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bda2e1125a485a9fd9fb1cb9aa220b. Latest consensus state: current_term: 1 leader_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } } }
I20260619 06:57:28.982947 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:28.983342 11631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:28.984501 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bda2e1125a485a9fd9fb1cb9aa220b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33657 } } }
I20260619 06:57:28.984637 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:28.987957 11631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:28.999660 11631 catalog_manager.cc:1347] Generated new cluster ID: 0856af3a592b455783773e4abeef625a
I20260619 06:57:28.999758 11631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:29.020856 11631 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:29.022569 11631 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:29.029274 11631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87bda2e1125a485a9fd9fb1cb9aa220b: Generated new TSK 0
I20260619 06:57:29.030062 11631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:29.037892 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11544
W20260619 06:57:29.045725 11529 connection.cc:535] client connection to 127.0.0.1:33657 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33657): Connection reset by peer (error 104)
2026-06-19T06:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:29 dist-test-slave-g8zg krb5kdc[11643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:29 dist-test-slave-g8zg krb5kdc[11643](info): set up 2 sockets
Jun 19 06:57:29 dist-test-slave-g8zg krb5kdc[11643](info): commencing operation
krb5kdc: starting...
W20260619 06:57:30.083879 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:30 dist-test-slave-g8zg krb5kdc[11643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:30.216383 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:30.325526 11656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:30.325867 11656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:30.325947 11656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:30.329988 11656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:30.330082 11656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:30.330120 11656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:30.330160 11656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:30.330210 11656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:30.335459 11656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:30.336519 11656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:30.337538 11656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:30.346364 11662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:30.346390 11664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:30.346390 11661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:30.346681 11656 server_base.cc:1034] running on GCE node
I20260619 06:57:30.347396 11656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:30.348551 11656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:30.349723 11656 hybrid_clock.cc:648] HybridClock initialized: now 1781852250349704 us; error 39 us; skew 500 ppm
Jun 19 06:57:30 dist-test-slave-g8zg krb5kdc[11643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:30.352617 11656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:30.353561 11656 webserver.cc:460] Webserver started at http://127.0.0.1:40841/ using document root <none> and password file <none>
I20260619 06:57:30.354135 11656 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:30.354228 11656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:30.354462 11656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:30.356542 11656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7"
format_stamp: "Formatted at 2026-06-19 06:57:30 on dist-test-slave-g8zg"
server_key: "32654b2083a2ebc426be9a03b9c1ae0c"
server_key_iv: "9bcfee5bb12c327a6a174745a3b975bc"
server_key_version: "encryptionkey@0"
I20260619 06:57:30.357105 11656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7"
format_stamp: "Formatted at 2026-06-19 06:57:30 on dist-test-slave-g8zg"
server_key: "32654b2083a2ebc426be9a03b9c1ae0c"
server_key_iv: "9bcfee5bb12c327a6a174745a3b975bc"
server_key_version: "encryptionkey@0"
I20260619 06:57:30.360989 11656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:30.363593 11671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:30.364970 11656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:30.365127 11656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7"
format_stamp: "Formatted at 2026-06-19 06:57:30 on dist-test-slave-g8zg"
server_key: "32654b2083a2ebc426be9a03b9c1ae0c"
server_key_iv: "9bcfee5bb12c327a6a174745a3b975bc"
server_key_version: "encryptionkey@0"
I20260619 06:57:30.365242 11656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:30.369886 11656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:30.372841 11656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:30.373040 11656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:30.380803 11656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39753
I20260619 06:57:30.380836 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39753 every 8 connection(s)
I20260619 06:57:30.381736 11656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:30.383208 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11656
I20260619 06:57:30.383396 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:30.383797 22980 external_mini_cluster.cc:1428] Setting key 184f610aa988c1ee0c94b02993eb8426
I20260619 06:57:30.385452 11724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:30.389423 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:30.385447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:51887 (local address 127.0.0.1:39753)
0619 06:57:30.385939 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:30.385956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:30.385995 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:57:30.386667 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:30.386684 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:30.386976 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:30.387359 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:30.388187 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:30.388192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:30.388203 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:30.388214 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:30.388266 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:30.388284 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:30.388442 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:30.388512 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:57:30.388684 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":168,"threads_started":1}
I20260619 06:57:30.392632 11724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7: Bootstrap starting.
I20260619 06:57:30.392728 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:30.395728 11724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:30.396760 11724 log.cc:824] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:30.397300 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:30.395301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:45895 (local address 127.0.0.1:39753)
0619 06:57:30.395463 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:30.395469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:30.395813 (+ 344us) server_negotiation.cc:407] Connection header received
0619 06:57:30.395946 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:30.395949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:30.395980 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:30.396053 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:30.396821 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:30.396824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:30.396826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:30.396836 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:30.396888 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:30.396907 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:30.397022 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:30.397124 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 06:57:30.397165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:57:30.399117 11724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7: No bootstrap required, opened a new log
I20260619 06:57:30.402714 11724 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260619 06:57:30.402948 11724 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:30.402984 11724 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2bbc30d7694cc6bbd56cc139f3e1c7, State: Initialized, Role: FOLLOWER
I20260619 06:57:30.403523 11724 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260619 06:57:30.403687 11724 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:30.403782 11724 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:30.403900 11724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:30.404839 11724 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260619 06:57:30.405153 11724 leader_election.cc:302] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb2bbc30d7694cc6bbd56cc139f3e1c7; no voters:
I20260619 06:57:30.405426 11724 leader_election.cc:288] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:30.405608 11734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:30.405920 11734 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [term 1 LEADER]: Becoming Leader. State: Replica: bb2bbc30d7694cc6bbd56cc139f3e1c7, State: Running, Role: LEADER
I20260619 06:57:30.406312 11734 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260619 06:57:30.406829 11724 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:30.408150 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } } }
I20260619 06:57:30.408310 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:30.408685 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2bbc30d7694cc6bbd56cc139f3e1c7. Latest consensus state: current_term: 1 leader_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2bbc30d7694cc6bbd56cc139f3e1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } } }
I20260619 06:57:30.408778 11743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:30.408807 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:30.412850 11743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:30.418938 11743 catalog_manager.cc:1347] Generated new cluster ID: a131d9e67789469e9cc50205f1b3074a
I20260619 06:57:30.419013 11743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:30.428315 11743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:30.428658 11743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:30.434717 11743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb2bbc30d7694cc6bbd56cc139f3e1c7: Generated new TSK 0
I20260619 06:57:30.435444 11743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:30.443602 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11656
2026-06-19T06:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:30 dist-test-slave-g8zg krb5kdc[11755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:30 dist-test-slave-g8zg krb5kdc[11755](info): set up 2 sockets
Jun 19 06:57:30 dist-test-slave-g8zg krb5kdc[11755](info): commencing operation
krb5kdc: starting...
W20260619 06:57:31.503847 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:31.639546 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:31.749663 11768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:31.750010 11768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:31.750131 11768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:31.757175 11768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:31.757278 11768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:31.757313 11768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:31.757352 11768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:31.757388 11768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:31.762890 11768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:31.764038 11768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:31.765048 11768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:31.774122 11774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:31.774122 11773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:31.774317 11776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:31.774792 11768 server_base.cc:1034] running on GCE node
I20260619 06:57:31.775457 11768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:31.776595 11768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:31.777860 11768 hybrid_clock.cc:648] HybridClock initialized: now 1781852251777836 us; error 48 us; skew 500 ppm
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:31.780944 11768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:31.781900 11768 webserver.cc:460] Webserver started at http://127.0.0.1:39503/ using document root <none> and password file <none>
I20260619 06:57:31.782476 11768 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:31.782567 11768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:31.782799 11768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:31.784871 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "8395eeff7c7c4c1cb78ae2a354308572"
format_stamp: "Formatted at 2026-06-19 06:57:31 on dist-test-slave-g8zg"
server_key: "9520ef770e143dc215a22f781be7dd7c"
server_key_iv: "aa16f5425a3aee75af8711dacd618bbc"
server_key_version: "encryptionkey@0"
I20260619 06:57:31.785425 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "8395eeff7c7c4c1cb78ae2a354308572"
format_stamp: "Formatted at 2026-06-19 06:57:31 on dist-test-slave-g8zg"
server_key: "9520ef770e143dc215a22f781be7dd7c"
server_key_iv: "aa16f5425a3aee75af8711dacd618bbc"
server_key_version: "encryptionkey@0"
I20260619 06:57:31.789356 11768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:57:31.792198 11783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:31.793641 11768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:31.793825 11768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "8395eeff7c7c4c1cb78ae2a354308572"
format_stamp: "Formatted at 2026-06-19 06:57:31 on dist-test-slave-g8zg"
server_key: "9520ef770e143dc215a22f781be7dd7c"
server_key_iv: "aa16f5425a3aee75af8711dacd618bbc"
server_key_version: "encryptionkey@0"
I20260619 06:57:31.793957 11768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:31.806190 11768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:31.809408 11768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:31.809657 11768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:31.817387 11768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37427
I20260619 06:57:31.817457 11835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37427 every 8 connection(s)
I20260619 06:57:31.818424 11768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:31.822345 11836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:31.827378 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11768
I20260619 06:57:31.827564 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:31.827886 22980 external_mini_cluster.cc:1428] Setting key bf0ac55d243e17e83f88055231cdf756
I20260619 06:57:31.830207 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: Bootstrap starting.
I20260619 06:57:31.833779 11836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:31.834892 11836 log.cc:824] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:31.837879 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: No bootstrap required, opened a new log
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:57:31.843020 11836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260619 06:57:31.843453 11836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:31.843657 11836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8395eeff7c7c4c1cb78ae2a354308572, State: Initialized, Role: FOLLOWER
I20260619 06:57:31.844403 11836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260619 06:57:31.844605 11836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:31.844695 11836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:31.844813 11836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:31.846339 11836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260619 06:57:31.846767 11836 leader_election.cc:302] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8395eeff7c7c4c1cb78ae2a354308572; no voters:
I20260619 06:57:31.847170 11836 leader_election.cc:288] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:31.847352 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:31.829703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:60143 (local address 127.0.0.1:37427)
0619 06:57:31.830237 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:31.830264 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:31.830321 (+ 57us) server_negotiation.cc:407] Connection header received
0619 06:57:31.831378 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:31.831406 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:31.831991 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:31.832451 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:31.833225 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:31.835004 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:31.836265 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:31.837052 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:31.837258 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:31.840204 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:31.840211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:31.840234 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:31.840270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:31.843325 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:31.844050 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:31.844057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:31.844063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:31.844227 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:31.844599 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:31.844603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:31.844606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:31.845246 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:31.845491 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:31.845827 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:57:31.846106 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
I20260619 06:57:31.847404 11841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:31.847714 11841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [term 1 LEADER]: Becoming Leader. State: Replica: 8395eeff7c7c4c1cb78ae2a354308572, State: Running, Role: LEADER
I20260619 06:57:31.848160 11841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } }
I20260619 06:57:31.848759 11836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:31.850492 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8395eeff7c7c4c1cb78ae2a354308572. Latest consensus state: current_term: 1 leader_uuid: "8395eeff7c7c4c1cb78ae2a354308572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260619 06:57:31.850638 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:31.851075 11842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8395eeff7c7c4c1cb78ae2a354308572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8395eeff7c7c4c1cb78ae2a354308572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37427 } } }
I20260619 06:57:31.851217 11842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:31.851289 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 06:57:31.855762 11856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:31.855916 11856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:31.856134 11851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:31.857241 11851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:31.863746 11851 catalog_manager.cc:1347] Generated new cluster ID: a71383dd88f040a58f1986f9e318f09e
I20260619 06:57:31.863812 11851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:31.866348 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:31.853628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:50227 (local address 127.0.0.1:37427)
0619 06:57:31.853758 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:31.853762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:31.855571 (+ 1809us) server_negotiation.cc:407] Connection header received
0619 06:57:31.855744 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:31.855746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:31.855794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:31.855882 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:31.856782 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:31.857732 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:31.858900 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:31.859360 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:31.859501 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:31.861308 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:31.861312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:31.861316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:31.861346 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:31.863668 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:31.864445 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:31.864449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:31.864452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:31.864548 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:31.865029 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:31.865033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:31.865035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:31.865288 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:31.865438 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:31.866004 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 06:57:31.866126 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:57:31.878253 11851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:31.879873 11851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:31.905666 11851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8395eeff7c7c4c1cb78ae2a354308572: Generated new TSK 0
I20260619 06:57:31.906844 11851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:31.916211 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11768
W20260619 06:57:31.923357 11753 connection.cc:535] client connection to 127.0.0.1:37427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37427): Connection reset by peer (error 104)
2026-06-19T06:57:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11867](info): set up 2 sockets
Jun 19 06:57:31 dist-test-slave-g8zg krb5kdc[11867](info): commencing operation
krb5kdc: starting...
W20260619 06:57:33.023564 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:33 dist-test-slave-g8zg krb5kdc[11867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:33.154237 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:33.278915 11880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:33.279297 11880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:33.279382 11880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:33.283459 11880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:33.283555 11880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:33.283591 11880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:33.283627 11880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:33.283666 11880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:33.289122 11880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:33.290436 11880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:33.291736 11880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:33.301862 11888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:33.301901 11885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:33.301901 11880 server_base.cc:1034] running on GCE node
W20260619 06:57:33.302047 11886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:33.302811 11880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:33.304144 11880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:33.305341 11880 hybrid_clock.cc:648] HybridClock initialized: now 1781852253305306 us; error 46 us; skew 500 ppm
Jun 19 06:57:33 dist-test-slave-g8zg krb5kdc[11867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:33.318331 11880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:33.319432 11880 webserver.cc:460] Webserver started at http://127.0.0.1:33193/ using document root <none> and password file <none>
I20260619 06:57:33.320080 11880 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:33.320174 11880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:33.320418 11880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:33.322554 11880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "3d45ebb65c914d3790e2fd99e83fa117"
format_stamp: "Formatted at 2026-06-19 06:57:33 on dist-test-slave-g8zg"
server_key: "5f9880d0c3ed6dc69cfcac9c54e4e7a4"
server_key_iv: "9abd1ff6c5ffeb5ea3d7467919f6fb3a"
server_key_version: "encryptionkey@0"
I20260619 06:57:33.323133 11880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "3d45ebb65c914d3790e2fd99e83fa117"
format_stamp: "Formatted at 2026-06-19 06:57:33 on dist-test-slave-g8zg"
server_key: "5f9880d0c3ed6dc69cfcac9c54e4e7a4"
server_key_iv: "9abd1ff6c5ffeb5ea3d7467919f6fb3a"
server_key_version: "encryptionkey@0"
I20260619 06:57:33.327178 11880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:57:33.329955 11895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:33.331455 11880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:57:33.331621 11880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "3d45ebb65c914d3790e2fd99e83fa117"
format_stamp: "Formatted at 2026-06-19 06:57:33 on dist-test-slave-g8zg"
server_key: "5f9880d0c3ed6dc69cfcac9c54e4e7a4"
server_key_iv: "9abd1ff6c5ffeb5ea3d7467919f6fb3a"
server_key_version: "encryptionkey@0"
I20260619 06:57:33.331740 11880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:33.355078 11880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:33.358228 11880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:33.358456 11880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:33.366025 11880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43363
I20260619 06:57:33.366053 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43363 every 8 connection(s)
I20260619 06:57:33.367022 11880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:33.370442 11948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:33.372412 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11880
I20260619 06:57:33.372531 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:33.372840 22980 external_mini_cluster.cc:1428] Setting key 75b2aafae9c747ecb6d686b67ececd8e
I20260619 06:57:33.378362 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117: Bootstrap starting.
I20260619 06:57:33.381491 11948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:33.382510 11948 log.cc:824] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:33.384049 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:33.374534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42969 (local address 127.0.0.1:43363)
0619 06:57:33.375045 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:33.375069 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:33.375135 (+ 66us) server_negotiation.cc:407] Connection header received
0619 06:57:33.376143 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:33.376170 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:33.376663 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:33.377093 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:33.377560 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:33.378997 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:33.380033 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:33.380729 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:33.380888 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:33.381918 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:33.381924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:33.381942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:33.382503 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:33.382577 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:33.382612 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:33.382749 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:33.382987 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:57:33.383146 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":139,"threads_started":1}
I20260619 06:57:33.385077 11948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117: No bootstrap required, opened a new log
I20260619 06:57:33.387213 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:33.389895 11948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260619 06:57:33.390220 11948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:33.390321 11948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d45ebb65c914d3790e2fd99e83fa117, State: Initialized, Role: FOLLOWER
I20260619 06:57:33.390833 11948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260619 06:57:33.390990 11948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:33.391067 11948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:33.391180 11948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:33.392467 11948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260619 06:57:33.392851 11948 leader_election.cc:302] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d45ebb65c914d3790e2fd99e83fa117; no voters:
I20260619 06:57:33.393250 11948 leader_election.cc:288] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:33.393375 11958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:33.393647 11958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [term 1 LEADER]: Becoming Leader. State: Replica: 3d45ebb65c914d3790e2fd99e83fa117, State: Running, Role: LEADER
I20260619 06:57:33.394120 11958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260619 06:57:33.394688 11948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:33.394898 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:33.389860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:49463 (local address 127.0.0.1:43363)
0619 06:57:33.389989 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:33.389994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:33.390268 (+ 274us) server_negotiation.cc:407] Connection header received
0619 06:57:33.390383 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:33.390388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:33.390444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:33.390537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:33.390838 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:33.391571 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:33.392726 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:33.393011 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:33.393161 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:33.394217 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:33.394222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:33.394225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:33.394241 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:33.394279 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:33.394294 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:33.394395 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:33.394715 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 06:57:33.394762 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:57:33.396291 11960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d45ebb65c914d3790e2fd99e83fa117. Latest consensus state: current_term: 1 leader_uuid: "3d45ebb65c914d3790e2fd99e83fa117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260619 06:57:33.396656 11960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:33.396554 11959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d45ebb65c914d3790e2fd99e83fa117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d45ebb65c914d3790e2fd99e83fa117" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260619 06:57:33.396850 11959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:33.397212 11972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:33.400440 11972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:33.405867 11972 catalog_manager.cc:1347] Generated new cluster ID: aea710cab8714fa5b3829264f12382f8
I20260619 06:57:33.405947 11972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:33.452373 11972 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:33.453526 11972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:33.458076 11972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d45ebb65c914d3790e2fd99e83fa117: Generated new TSK 0
I20260619 06:57:33.458843 11972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:33.463711 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11880
W20260619 06:57:33.469525 11865 connection.cc:535] client connection to 127.0.0.1:43363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43363): Connection reset by peer (error 104)
2026-06-19T06:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:33 dist-test-slave-g8zg krb5kdc[11979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:33 dist-test-slave-g8zg krb5kdc[11979](info): set up 2 sockets
Jun 19 06:57:33 dist-test-slave-g8zg krb5kdc[11979](info): commencing operation
krb5kdc: starting...
W20260619 06:57:34.511847 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:34 dist-test-slave-g8zg krb5kdc[11979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:57:34.645888 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:34.760555 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:34.760830 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:34.760872 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:34.764755 11992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:34.764814 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:34.764833 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:34.764850 11992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:34.764866 11992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:34.770053 11992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:34.771077 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:34.772078 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:34.781158 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:34.781165 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:34.781339 11992 server_base.cc:1034] running on GCE node
W20260619 06:57:34.781556 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:34.782305 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:34.783852 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:34.785024 11992 hybrid_clock.cc:648] HybridClock initialized: now 1781852254785003 us; error 40 us; skew 500 ppm
Jun 19 06:57:34 dist-test-slave-g8zg krb5kdc[11979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:34.788161 11992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:57:34.789212 11992 webserver.cc:460] Webserver started at http://127.0.0.1:42425/ using document root <none> and password file <none>
I20260619 06:57:34.789883 11992 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:34.789979 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:34.790238 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:34.792673 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "2d57ca6444184b028a981ea25532a6a1"
format_stamp: "Formatted at 2026-06-19 06:57:34 on dist-test-slave-g8zg"
server_key: "8f752302871a8bf82ba747f481935cee"
server_key_iv: "ca454e3240373bfb4a924589fa61a3b7"
server_key_version: "encryptionkey@0"
I20260619 06:57:34.793314 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "2d57ca6444184b028a981ea25532a6a1"
format_stamp: "Formatted at 2026-06-19 06:57:34 on dist-test-slave-g8zg"
server_key: "8f752302871a8bf82ba747f481935cee"
server_key_iv: "ca454e3240373bfb4a924589fa61a3b7"
server_key_version: "encryptionkey@0"
I20260619 06:57:34.797866 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:34.800841 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:34.802397 11992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:34.802564 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "2d57ca6444184b028a981ea25532a6a1"
format_stamp: "Formatted at 2026-06-19 06:57:34 on dist-test-slave-g8zg"
server_key: "8f752302871a8bf82ba747f481935cee"
server_key_iv: "ca454e3240373bfb4a924589fa61a3b7"
server_key_version: "encryptionkey@0"
I20260619 06:57:34.802716 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:34.808085 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:34.811507 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:34.811717 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:34.820503 11992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44695
I20260619 06:57:34.820591 12059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44695 every 8 connection(s)
I20260619 06:57:34.821659 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:34.823181 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 11992
I20260619 06:57:34.823416 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:34.823786 22980 external_mini_cluster.cc:1428] Setting key a55f0928ad30a1d2018d6ddeabb976c4
I20260619 06:57:34.825341 12060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:34.830327 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:34.825539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39895 (local address 127.0.0.1:44695)
0619 06:57:34.826067 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:34.826087 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:34.826137 (+ 50us) server_negotiation.cc:407] Connection header received
0619 06:57:34.827047 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:34.827068 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:34.827452 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:34.827831 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:34.828692 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:34.828698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:34.828712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:34.828725 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:34.828789 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:34.828817 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:34.828991 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:34.829061 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:57:34.829414 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260619 06:57:34.832358 12060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1: Bootstrap starting.
I20260619 06:57:34.833750 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:34.834931 12060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:34.835752 12060 log.cc:824] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:34.838075 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:34.836188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:37183 (local address 127.0.0.1:44695)
0619 06:57:34.836334 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:34.836339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:34.836533 (+ 194us) server_negotiation.cc:407] Connection header received
0619 06:57:34.836630 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:34.836635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:34.836676 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:34.836775 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:34.837596 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:34.837599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:34.837601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:34.837611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:34.837639 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:34.837649 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:34.837726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:34.837849 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 06:57:34.837887 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:57:34.838347 12060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1: No bootstrap required, opened a new log
I20260619 06:57:34.841878 12060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260619 06:57:34.842095 12060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:34.842157 12060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d57ca6444184b028a981ea25532a6a1, State: Initialized, Role: FOLLOWER
I20260619 06:57:34.842619 12060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260619 06:57:34.842743 12060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:34.842831 12060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:34.842942 12060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:34.843935 12060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260619 06:57:34.844274 12060 leader_election.cc:302] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d57ca6444184b028a981ea25532a6a1; no voters:
I20260619 06:57:34.844640 12060 leader_election.cc:288] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:34.844810 12070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:34.845110 12070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d57ca6444184b028a981ea25532a6a1, State: Running, Role: LEADER
I20260619 06:57:34.845510 12070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260619 06:57:34.845911 12060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:34.847645 12071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d57ca6444184b028a981ea25532a6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260619 06:57:34.847797 12071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:34.848174 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d57ca6444184b028a981ea25532a6a1. Latest consensus state: current_term: 1 leader_uuid: "2d57ca6444184b028a981ea25532a6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57ca6444184b028a981ea25532a6a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260619 06:57:34.848282 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:34.848320 12082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:34.851157 12082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:34.857571 12082 catalog_manager.cc:1347] Generated new cluster ID: c2a7f8e98a0346069f0cf0e5939fc5b2
I20260619 06:57:34.857650 12082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:34.896111 12082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:34.896543 12082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:34.904639 12082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d57ca6444184b028a981ea25532a6a1: Generated new TSK 0
I20260619 06:57:34.905313 12082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:34.908670 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 11992
2026-06-19T06:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:34 dist-test-slave-g8zg krb5kdc[12091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:34 dist-test-slave-g8zg krb5kdc[12091](info): set up 2 sockets
Jun 19 06:57:34 dist-test-slave-g8zg krb5kdc[12091](info): commencing operation
krb5kdc: starting...
W20260619 06:57:35.955758 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:36.086705 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38827
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:36.212636 12104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:36.212949 12104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:36.213032 12104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:36.217156 12104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:36.217223 12104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:36.217244 12104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:36.217262 12104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:36.217278 12104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:36.222785 12104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:38827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:36.224053 12104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:36.225292 12104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:36.235177 12109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:36.235188 12112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:36.235136 12110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:36.235766 12104 server_base.cc:1034] running on GCE node
I20260619 06:57:36.236410 12104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:36.237795 12104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:36.239032 12104 hybrid_clock.cc:648] HybridClock initialized: now 1781852256239009 us; error 64 us; skew 500 ppm
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852256, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:36.242030 12104 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:36.242926 12104 webserver.cc:460] Webserver started at http://127.22.113.62:43309/ using document root <none> and password file <none>
I20260619 06:57:36.243577 12104 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:36.243643 12104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:36.243882 12104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:36.246001 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "8231f2294a3846a688ca67074bf1dc77"
format_stamp: "Formatted at 2026-06-19 06:57:36 on dist-test-slave-g8zg"
server_key: "779316282e31292c76c30e352cf00a3b"
server_key_iv: "76761db2f97d655550b31517493310b2"
server_key_version: "encryptionkey@0"
I20260619 06:57:36.246579 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "8231f2294a3846a688ca67074bf1dc77"
format_stamp: "Formatted at 2026-06-19 06:57:36 on dist-test-slave-g8zg"
server_key: "779316282e31292c76c30e352cf00a3b"
server_key_iv: "76761db2f97d655550b31517493310b2"
server_key_version: "encryptionkey@0"
I20260619 06:57:36.250891 12104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:57:36.253865 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:36.255566 12104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:36.255730 12104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "8231f2294a3846a688ca67074bf1dc77"
format_stamp: "Formatted at 2026-06-19 06:57:36 on dist-test-slave-g8zg"
server_key: "779316282e31292c76c30e352cf00a3b"
server_key_iv: "76761db2f97d655550b31517493310b2"
server_key_version: "encryptionkey@0"
I20260619 06:57:36.255857 12104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:36.276712 12104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:36.279803 12104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:36.280019 12104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:36.287986 12104 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:38827
I20260619 06:57:36.288008 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:38827 every 8 connection(s)
I20260619 06:57:36.289017 12104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:36.292448 12172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:36.294895 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12104
I20260619 06:57:36.295065 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:36.295441 22980 external_mini_cluster.cc:1428] Setting key 5db93c02041b03065ce9241f06da2011
I20260619 06:57:36.300007 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77: Bootstrap starting.
I20260619 06:57:36.302691 12172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:36.303783 12172 log.cc:824] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:36.306274 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77: No bootstrap required, opened a new log
I20260619 06:57:36.311657 12172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } }
I20260619 06:57:36.311959 12172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:36.312037 12172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8231f2294a3846a688ca67074bf1dc77, State: Initialized, Role: FOLLOWER
I20260619 06:57:36.312572 12172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } }
I20260619 06:57:36.312718 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:36.297259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56829 (local address 127.22.113.62:38827)
0619 06:57:36.297822 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:36.297840 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:36.297895 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:57:36.299071 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:36.299096 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:36.299635 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:36.300019 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:36.300916 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:36.302013 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:36.302946 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:36.303786 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:36.303996 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:36.306779 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:36.306785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:36.306804 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:36.306838 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:36.309181 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:36.309749 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:36.309756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:36.309762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:36.309917 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:36.310241 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:36.310244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:36.310245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:36.310826 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:36.311052 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:36.311323 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:57:36.311572 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":176,"threads_started":1}
I20260619 06:57:36.312757 12172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:36.312816 12172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:36.312903 12172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:36.314216 12172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } }
I20260619 06:57:36.314604 12172 leader_election.cc:302] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8231f2294a3846a688ca67074bf1dc77; no voters:
I20260619 06:57:36.315022 12172 leader_election.cc:288] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:36.315204 12177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:36.315577 12177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [term 1 LEADER]: Becoming Leader. State: Replica: 8231f2294a3846a688ca67074bf1dc77, State: Running, Role: LEADER
I20260619 06:57:36.316078 12177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } }
I20260619 06:57:36.316558 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:36.316638 12172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:36.318135 12178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8231f2294a3846a688ca67074bf1dc77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } } }
I20260619 06:57:36.318274 12178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:36.318854 12191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:36.318840 12179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8231f2294a3846a688ca67074bf1dc77. Latest consensus state: current_term: 1 leader_uuid: "8231f2294a3846a688ca67074bf1dc77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8231f2294a3846a688ca67074bf1dc77" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 38827 } } }
I20260619 06:57:36.320708 12179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:36.322854 12191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:36.328631 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:36.319829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36749 (local address 127.22.113.62:38827)
0619 06:57:36.319975 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:36.319982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:36.320431 (+ 449us) server_negotiation.cc:407] Connection header received
0619 06:57:36.320622 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:36.320627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:36.320677 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:36.320779 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:36.321504 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:36.322070 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:36.323019 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:36.323330 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:36.323445 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:36.325005 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:36.325009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:36.325012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:36.325035 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:36.326521 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:36.327001 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:36.327005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:36.327006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:36.327118 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:36.327548 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:36.327552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:36.327553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:36.327782 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:36.327878 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:36.328307 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 06:57:36.328411 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:57:36.330013 12191 catalog_manager.cc:1347] Generated new cluster ID: 8e6050f2e23343089d63929e0c244486
I20260619 06:57:36.330144 12191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:36.362213 12191 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:36.363459 12191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:36.369674 12191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8231f2294a3846a688ca67074bf1dc77: Generated new TSK 0
I20260619 06:57:36.370546 12191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:36.379699 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12104
W20260619 06:57:36.386404 12089 connection.cc:535] client connection to 127.22.113.62:38827 recv error: Network error: failed to read from TLS socket (remote: 127.22.113.62:38827): Connection reset by peer (error 104)
2026-06-19T06:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12203](info): set up 2 sockets
Jun 19 06:57:36 dist-test-slave-g8zg krb5kdc[12203](info): commencing operation
krb5kdc: starting...
W20260619 06:57:37.423683 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:37 dist-test-slave-g8zg krb5kdc[12203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:37.559123 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41563
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:41563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:37.677044 12216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:37.677313 12216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:37.677354 12216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:37.681329 12216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:37.681393 12216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:37.681413 12216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:37.681430 12216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:37.681447 12216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:37.686633 12216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:41563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:37.687717 12216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:37.688691 12216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:37.697875 12221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:37.697947 12222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:37.698037 12216 server_base.cc:1034] running on GCE node
W20260619 06:57:37.697950 12224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:37.698904 12216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:37.700347 12216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:37.701562 12216 hybrid_clock.cc:648] HybridClock initialized: now 1781852257701536 us; error 40 us; skew 500 ppm
Jun 19 06:57:37 dist-test-slave-g8zg krb5kdc[12203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852257, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:37.704689 12216 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:37.705778 12216 webserver.cc:460] Webserver started at http://127.22.113.62:36599/ using document root <none> and password file <none>
I20260619 06:57:37.706450 12216 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:37.706518 12216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:37.706758 12216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:37.709214 12216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "b28d82062b394cf4a8fadf48f95fc2fc"
format_stamp: "Formatted at 2026-06-19 06:57:37 on dist-test-slave-g8zg"
server_key: "601cf1b0f096fc8475e0e8c899a0297f"
server_key_iv: "6a8a1abe87c055ede6562e64d200f66d"
server_key_version: "encryptionkey@0"
I20260619 06:57:37.709852 12216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "b28d82062b394cf4a8fadf48f95fc2fc"
format_stamp: "Formatted at 2026-06-19 06:57:37 on dist-test-slave-g8zg"
server_key: "601cf1b0f096fc8475e0e8c899a0297f"
server_key_iv: "6a8a1abe87c055ede6562e64d200f66d"
server_key_version: "encryptionkey@0"
I20260619 06:57:37.714360 12216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:57:37.717243 12231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:37.718770 12216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:37.718922 12216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "b28d82062b394cf4a8fadf48f95fc2fc"
format_stamp: "Formatted at 2026-06-19 06:57:37 on dist-test-slave-g8zg"
server_key: "601cf1b0f096fc8475e0e8c899a0297f"
server_key_iv: "6a8a1abe87c055ede6562e64d200f66d"
server_key_version: "encryptionkey@0"
I20260619 06:57:37.719044 12216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:37.740125 12216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:37.743819 12216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:37.744052 12216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:37.753602 12216 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:41563
I20260619 06:57:37.753688 12283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:41563 every 8 connection(s)
I20260619 06:57:37.755182 12216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:37.758492 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:37.762732 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12216
I20260619 06:57:37.762920 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:37.763235 22980 external_mini_cluster.cc:1428] Setting key 4a36db9adabcd6ae5fcac2e2b38a0355
I20260619 06:57:37.764376 12284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc: Bootstrap starting.
I20260619 06:57:37.766708 12284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:37.767542 12284 log.cc:824] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:37.769436 12284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc: No bootstrap required, opened a new log
I20260619 06:57:37.773404 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:37.764898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34343 (local address 127.22.113.62:41563)
0619 06:57:37.765408 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:37.765428 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:37.765472 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:57:37.766265 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:37.766285 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:37.766718 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:37.767161 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:37.767722 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:37.768791 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:37.769780 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:37.770291 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:37.770481 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:37.771499 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:37.771504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:37.771518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:37.771963 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:37.772024 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:37.772047 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:37.772195 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:37.772374 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 06:57:37.772524 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":157,"threads_started":1}
I20260619 06:57:37.773502 12284 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } }
I20260619 06:57:37.773799 12284 raft_consensus.cc:374] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:37.773849 12284 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28d82062b394cf4a8fadf48f95fc2fc, State: Initialized, Role: FOLLOWER
I20260619 06:57:37.774304 12284 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } }
I20260619 06:57:37.774466 12284 raft_consensus.cc:388] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:37.774590 12284 raft_consensus.cc:482] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:37.774714 12284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:37.775820 12284 raft_consensus.cc:504] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } }
I20260619 06:57:37.776189 12284 leader_election.cc:302] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b28d82062b394cf4a8fadf48f95fc2fc; no voters:
I20260619 06:57:37.776577 12284 leader_election.cc:288] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:37.776623 12289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:37.776839 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:37.776927 12289 raft_consensus.cc:686] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [term 1 LEADER]: Becoming Leader. State: Replica: b28d82062b394cf4a8fadf48f95fc2fc, State: Running, Role: LEADER
I20260619 06:57:37.777223 12289 consensus_queue.cc:237] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } }
I20260619 06:57:37.777917 12284 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:37.778396 12289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b28d82062b394cf4a8fadf48f95fc2fc. Latest consensus state: current_term: 1 leader_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } } }
I20260619 06:57:37.778584 12289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:37.778997 12291 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28d82062b394cf4a8fadf48f95fc2fc" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 41563 } } }
I20260619 06:57:37.779100 12291 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:37.779487 12297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:37.783488 12297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:37.785844 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:37.781050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49551 (local address 127.22.113.62:41563)
0619 06:57:37.781164 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:37.781168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:37.781430 (+ 262us) server_negotiation.cc:407] Connection header received
0619 06:57:37.781560 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:37.781563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:37.781608 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:37.781702 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:37.781991 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:37.782536 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:37.783966 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:37.784246 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:37.784308 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:37.785351 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:37.785355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:37.785357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:37.785371 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:37.785404 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:37.785415 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:37.785516 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:37.785642 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 06:57:37.785677 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:57:37.789757 12297 catalog_manager.cc:1347] Generated new cluster ID: 0d0b91c45c6b486b85892ca9863683e4
I20260619 06:57:37.789906 12297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:37.808668 12297 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:37.809944 12297 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:37.821846 12297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b28d82062b394cf4a8fadf48f95fc2fc: Generated new TSK 0
I20260619 06:57:37.822916 12297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:37.831281 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12216
W20260619 06:57:37.837299 12201 connection.cc:535] client connection to 127.22.113.62:41563 recv error: Network error: failed to read from TLS socket (remote: 127.22.113.62:41563): Connection reset by peer (error 104)
2026-06-19T06:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:37 dist-test-slave-g8zg krb5kdc[12315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:37 dist-test-slave-g8zg krb5kdc[12315](info): set up 2 sockets
Jun 19 06:57:37 dist-test-slave-g8zg krb5kdc[12315](info): commencing operation
krb5kdc: starting...
W20260619 06:57:38.875622 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:38 dist-test-slave-g8zg krb5kdc[12315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:39.007114 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42353
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:39.139498 12328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:39.139807 12328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:39.139851 12328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:39.144012 12328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:39.144088 12328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:39.144109 12328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:39.144125 12328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:39.144141 12328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:39.149603 12328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:42353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:39.150707 12328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:39.151811 12328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:39.161350 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:39.161332 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:39.161327 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:39.161662 12328 server_base.cc:1034] running on GCE node
I20260619 06:57:39.162297 12328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:39.163579 12328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:39.164772 12328 hybrid_clock.cc:648] HybridClock initialized: now 1781852259164738 us; error 57 us; skew 500 ppm
Jun 19 06:57:39 dist-test-slave-g8zg krb5kdc[12315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852259, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:39.167865 12328 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:39.168929 12328 webserver.cc:460] Webserver started at http://127.22.113.62:42329/ using document root <none> and password file <none>
I20260619 06:57:39.169529 12328 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:39.169620 12328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:39.169842 12328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:39.172020 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "e7f8718c430b4ba9ae38c38781c3dc4f"
format_stamp: "Formatted at 2026-06-19 06:57:39 on dist-test-slave-g8zg"
server_key: "bbc8ba5e77ab225a15836c1e0bb07a05"
server_key_iv: "bcd10c7f489b6f702c4501df7a536ab7"
server_key_version: "encryptionkey@0"
I20260619 06:57:39.172595 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "e7f8718c430b4ba9ae38c38781c3dc4f"
format_stamp: "Formatted at 2026-06-19 06:57:39 on dist-test-slave-g8zg"
server_key: "bbc8ba5e77ab225a15836c1e0bb07a05"
server_key_iv: "bcd10c7f489b6f702c4501df7a536ab7"
server_key_version: "encryptionkey@0"
I20260619 06:57:39.176745 12328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:39.179543 12343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:39.180950 12328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:39.181102 12328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "e7f8718c430b4ba9ae38c38781c3dc4f"
format_stamp: "Formatted at 2026-06-19 06:57:39 on dist-test-slave-g8zg"
server_key: "bbc8ba5e77ab225a15836c1e0bb07a05"
server_key_iv: "bcd10c7f489b6f702c4501df7a536ab7"
server_key_version: "encryptionkey@0"
I20260619 06:57:39.181216 12328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:39.185974 12328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:39.189180 12328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:39.189388 12328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:39.197080 12328 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:42353
I20260619 06:57:39.197140 12395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:42353 every 8 connection(s)
I20260619 06:57:39.198062 12328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:39.201584 12396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:39.204634 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12328
I20260619 06:57:39.204813 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:39.205111 22980 external_mini_cluster.cc:1428] Setting key 91e290745d8108703fa94634219a502f
I20260619 06:57:39.208433 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f: Bootstrap starting.
I20260619 06:57:39.211058 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:39.206671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33589 (local address 127.22.113.62:42353)
0619 06:57:39.207238 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:39.207320 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:39.207383 (+ 63us) server_negotiation.cc:407] Connection header received
0619 06:57:39.208195 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:39.208214 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:39.208548 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:39.208971 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:39.209779 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:39.209784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:39.209794 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:39.209804 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:39.209861 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:39.209881 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:39.210063 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:39.210136 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:57:39.210296 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":169,"threads_started":1}
I20260619 06:57:39.211248 12396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:39.212152 12396 log.cc:824] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:39.214426 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:39.214746 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f: No bootstrap required, opened a new log
I20260619 06:57:39.218676 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:39.216741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42243 (local address 127.22.113.62:42353)
0619 06:57:39.216863 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:39.216869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:39.217088 (+ 219us) server_negotiation.cc:407] Connection header received
0619 06:57:39.217230 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:39.217234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:39.217274 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:39.217401 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:39.218249 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:39.218252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:39.218255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:39.218266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:39.218291 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:39.218311 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:39.218407 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:39.218505 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 06:57:39.218543 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:57:39.219511 12396 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } }
I20260619 06:57:39.219833 12396 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:39.219902 12396 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f8718c430b4ba9ae38c38781c3dc4f, State: Initialized, Role: FOLLOWER
I20260619 06:57:39.220361 12396 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } }
I20260619 06:57:39.220490 12396 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:39.220533 12396 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:39.220614 12396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:39.221896 12396 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } }
I20260619 06:57:39.222266 12396 leader_election.cc:302] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7f8718c430b4ba9ae38c38781c3dc4f; no voters:
I20260619 06:57:39.222613 12396 leader_election.cc:288] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:39.222820 12406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:39.223174 12406 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [term 1 LEADER]: Becoming Leader. State: Replica: e7f8718c430b4ba9ae38c38781c3dc4f, State: Running, Role: LEADER
I20260619 06:57:39.223636 12406 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } }
I20260619 06:57:39.224095 12396 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:39.225572 12408 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f8718c430b4ba9ae38c38781c3dc4f. Latest consensus state: current_term: 1 leader_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } } }
I20260619 06:57:39.225729 12408 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:39.225922 12407 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f8718c430b4ba9ae38c38781c3dc4f" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42353 } } }
I20260619 06:57:39.226056 12407 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:39.226143 12416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:39.229566 12416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:39.236123 12416 catalog_manager.cc:1347] Generated new cluster ID: cd76b95c3c6e478c910ab026e9de84c8
I20260619 06:57:39.236207 12416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:39.253010 12416 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:39.253484 12416 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:39.272452 12416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7f8718c430b4ba9ae38c38781c3dc4f: Generated new TSK 0
I20260619 06:57:39.273357 12416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:39.284253 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12328
2026-06-19T06:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:39 dist-test-slave-g8zg krb5kdc[12427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:39 dist-test-slave-g8zg krb5kdc[12427](info): set up 2 sockets
Jun 19 06:57:39 dist-test-slave-g8zg krb5kdc[12427](info): commencing operation
krb5kdc: starting...
W20260619 06:57:40.320127 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:40.457062 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35669
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:40.566869 12440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:40.567183 12440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:40.567291 12440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:40.571238 12440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:40.571386 12440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:40.571430 12440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:40.571466 12440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:40.571489 12440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:40.576822 12440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:40.577890 12440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:40.578879 12440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:40.587864 12448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:40.587951 12445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:40.587864 12446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:40.588400 12440 server_base.cc:1034] running on GCE node
I20260619 06:57:40.588928 12440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:40.590040 12440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:40.591212 12440 hybrid_clock.cc:648] HybridClock initialized: now 1781852260591191 us; error 39 us; skew 500 ppm
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852260, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:40.594127 12440 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:40.595052 12440 webserver.cc:460] Webserver started at http://127.22.113.62:42127/ using document root <none> and password file <none>
I20260619 06:57:40.595665 12440 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:40.595731 12440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:40.595985 12440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:40.598002 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "16b6a05eeeb44e7d95ede7aca287c7c3"
format_stamp: "Formatted at 2026-06-19 06:57:40 on dist-test-slave-g8zg"
server_key: "6e0b60d093edc5760173b71571f9f5aa"
server_key_iv: "0f15f601003e43c06184e9661420315d"
server_key_version: "encryptionkey@0"
I20260619 06:57:40.598630 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "16b6a05eeeb44e7d95ede7aca287c7c3"
format_stamp: "Formatted at 2026-06-19 06:57:40 on dist-test-slave-g8zg"
server_key: "6e0b60d093edc5760173b71571f9f5aa"
server_key_iv: "0f15f601003e43c06184e9661420315d"
server_key_version: "encryptionkey@0"
I20260619 06:57:40.602484 12440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:57:40.605185 12455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:40.606557 12440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:57:40.606719 12440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "16b6a05eeeb44e7d95ede7aca287c7c3"
format_stamp: "Formatted at 2026-06-19 06:57:40 on dist-test-slave-g8zg"
server_key: "6e0b60d093edc5760173b71571f9f5aa"
server_key_iv: "0f15f601003e43c06184e9661420315d"
server_key_version: "encryptionkey@0"
I20260619 06:57:40.606840 12440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:40.618386 12440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:40.621187 12440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:40.621392 12440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:40.628893 12440 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:35669
I20260619 06:57:40.628952 12507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:35669 every 8 connection(s)
I20260619 06:57:40.629877 12440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:40.633256 12508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:40.634377 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12440
I20260619 06:57:40.634536 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:40.634903 22980 external_mini_cluster.cc:1428] Setting key 44214afab9c7ef5c2b599d3f5bd3df80
I20260619 06:57:40.640313 12508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3: Bootstrap starting.
I20260619 06:57:40.642805 12508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:40.643774 12508 log.cc:824] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:40.646728 12508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3: No bootstrap required, opened a new log
I20260619 06:57:40.651444 12508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } }
I20260619 06:57:40.651731 12508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:40.651822 12508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b6a05eeeb44e7d95ede7aca287c7c3, State: Initialized, Role: FOLLOWER
I20260619 06:57:40.652379 12508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } }
I20260619 06:57:40.652519 12508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:40.652621 12508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:40.652769 12508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:40.653084 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:40.636674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39995 (local address 127.22.113.62:35669)
0619 06:57:40.637239 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:40.637257 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:40.637298 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:57:40.638174 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:40.638193 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:40.638586 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:40.638978 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:40.639932 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:40.641078 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:40.642018 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:40.642545 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:40.642734 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:40.646338 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:40.646349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:40.646372 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:40.646417 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:40.649560 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:40.650099 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:40.650106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:40.650120 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:40.650243 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:40.650531 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:40.650534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:40.650536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:40.651024 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:40.651299 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:40.651591 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:57:40.651991 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":176,"threads_started":1}
I20260619 06:57:40.653970 12508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } }
I20260619 06:57:40.654368 12508 leader_election.cc:302] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b6a05eeeb44e7d95ede7aca287c7c3; no voters:
I20260619 06:57:40.654757 12508 leader_election.cc:288] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:40.654896 12513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:40.655143 12513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [term 1 LEADER]: Becoming Leader. State: Replica: 16b6a05eeeb44e7d95ede7aca287c7c3, State: Running, Role: LEADER
I20260619 06:57:40.655627 12513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } }
I20260619 06:57:40.656255 12508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:40.656579 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:40.657658 12515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b6a05eeeb44e7d95ede7aca287c7c3. Latest consensus state: current_term: 1 leader_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } } }
I20260619 06:57:40.657824 12515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:40.658221 12514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6a05eeeb44e7d95ede7aca287c7c3" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35669 } } }
I20260619 06:57:40.658326 12514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:40.658335 12524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:40.661790 12524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:40.668687 12524 catalog_manager.cc:1347] Generated new cluster ID: 066a01e922c64bb59c0264a10ef7017b
I20260619 06:57:40.668861 12524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:40.673352 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:40.659340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40931 (local address 127.22.113.62:35669)
0619 06:57:40.659483 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:40.659488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:40.663652 (+ 4164us) server_negotiation.cc:407] Connection header received
0619 06:57:40.663861 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:40.663865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:40.663926 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:40.664036 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:40.664978 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:40.665785 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:40.667041 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:40.667469 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:40.667594 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:40.669407 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:40.669411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:40.669414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:40.669443 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:40.671462 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:40.671975 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:40.671978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:40.671980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:40.672070 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:40.672388 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:40.672391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:40.672393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:40.672598 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:40.672733 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:40.673023 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:57:40.673164 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:57:40.693979 12524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:40.695042 12524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:40.705964 12524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b6a05eeeb44e7d95ede7aca287c7c3: Generated new TSK 0
I20260619 06:57:40.706898 12524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:40.716791 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12440
2026-06-19T06:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12539](info): set up 2 sockets
Jun 19 06:57:40 dist-test-slave-g8zg krb5kdc[12539](info): commencing operation
krb5kdc: starting...
W20260619 06:57:41.769375 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:41 dist-test-slave-g8zg krb5kdc[12539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:41.906378 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42443
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:42.017205 12552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:42.017539 12552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:42.017618 12552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:42.021756 12552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:42.021852 12552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:42.021890 12552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:42.021950 12552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:42.021986 12552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:42.027221 12552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:42.028345 12552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:42.029376 12552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:42.038453 12560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:42.038494 12557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:42.038681 12552 server_base.cc:1034] running on GCE node
W20260619 06:57:42.039072 12558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:42.039891 12552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:42.041306 12552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:42.042486 12552 hybrid_clock.cc:648] HybridClock initialized: now 1781852262042462 us; error 43 us; skew 500 ppm
Jun 19 06:57:42 dist-test-slave-g8zg krb5kdc[12539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852262, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:42.045855 12552 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:42.046948 12552 webserver.cc:460] Webserver started at http://127.22.113.62:43273/ using document root <none> and password file <none>
I20260619 06:57:42.047706 12552 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:42.047777 12552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:42.048022 12552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:42.050438 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "0580eaeeecee43b8bb3a86fcb1b84d31"
format_stamp: "Formatted at 2026-06-19 06:57:42 on dist-test-slave-g8zg"
server_key: "cd4540d8cb9e786e7f1fe2f56dffe8d0"
server_key_iv: "403646fa125e63973546da0a0c79df97"
server_key_version: "encryptionkey@0"
I20260619 06:57:42.051095 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "0580eaeeecee43b8bb3a86fcb1b84d31"
format_stamp: "Formatted at 2026-06-19 06:57:42 on dist-test-slave-g8zg"
server_key: "cd4540d8cb9e786e7f1fe2f56dffe8d0"
server_key_iv: "403646fa125e63973546da0a0c79df97"
server_key_version: "encryptionkey@0"
I20260619 06:57:42.055699 12552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:57:42.058615 12567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:42.060269 12552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:42.060433 12552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "0580eaeeecee43b8bb3a86fcb1b84d31"
format_stamp: "Formatted at 2026-06-19 06:57:42 on dist-test-slave-g8zg"
server_key: "cd4540d8cb9e786e7f1fe2f56dffe8d0"
server_key_iv: "403646fa125e63973546da0a0c79df97"
server_key_version: "encryptionkey@0"
I20260619 06:57:42.060556 12552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:42.081351 12552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:42.084898 12552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:42.085129 12552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:42.093806 12552 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:42443
I20260619 06:57:42.093868 12619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:42443 every 8 connection(s)
I20260619 06:57:42.094990 12552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:42.098037 12620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:42.103628 12620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31: Bootstrap starting.
I20260619 06:57:42.104318 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12552
I20260619 06:57:42.104471 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:42.104768 22980 external_mini_cluster.cc:1428] Setting key e76f6af2e1b452445535c8df47d5c2fa
I20260619 06:57:42.106490 12620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:42.107496 12620 log.cc:824] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:42.109616 12620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31: No bootstrap required, opened a new log
I20260619 06:57:42.113334 12620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } }
I20260619 06:57:42.113555 12620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:42.113588 12620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0580eaeeecee43b8bb3a86fcb1b84d31, State: Initialized, Role: FOLLOWER
I20260619 06:57:42.114076 12620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } }
I20260619 06:57:42.114230 12620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:42.114295 12620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:42.114430 12620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:42.115370 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:42.106126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48371 (local address 127.22.113.62:42443)
0619 06:57:42.106660 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:42.106688 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:42.106749 (+ 61us) server_negotiation.cc:407] Connection header received
0619 06:57:42.107969 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:42.108002 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:42.108520 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:42.108919 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:42.109398 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:42.110512 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:42.111509 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:42.112026 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:42.112224 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:42.113154 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:42.113160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:42.113181 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:42.113713 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:42.113787 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:42.113813 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:42.113921 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:42.114079 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:57:42.114346 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
I20260619 06:57:42.115396 12620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } }
I20260619 06:57:42.115800 12620 leader_election.cc:302] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0580eaeeecee43b8bb3a86fcb1b84d31; no voters:
I20260619 06:57:42.116125 12620 leader_election.cc:288] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:42.116276 12625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:42.116570 12625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [term 1 LEADER]: Becoming Leader. State: Replica: 0580eaeeecee43b8bb3a86fcb1b84d31, State: Running, Role: LEADER
I20260619 06:57:42.116956 12625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } }
I20260619 06:57:42.117408 12620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:42.118791 12627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0580eaeeecee43b8bb3a86fcb1b84d31. Latest consensus state: current_term: 1 leader_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } } }
I20260619 06:57:42.118947 12627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:42.119091 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:42.119405 12634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:42.119395 12626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0580eaeeecee43b8bb3a86fcb1b84d31" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 42443 } } }
I20260619 06:57:42.119527 12626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:42.123769 12634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:42.125741 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:42.121692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36651 (local address 127.22.113.62:42443)
0619 06:57:42.121794 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:42.121798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:42.122003 (+ 205us) server_negotiation.cc:407] Connection header received
0619 06:57:42.122222 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:42.122226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:42.122280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:42.122368 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:42.122645 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:42.123184 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:42.124127 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:42.124359 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:42.124472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:42.125306 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:42.125310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:42.125313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:42.125325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:42.125361 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:42.125372 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:42.125452 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:42.125545 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:57:42.125598 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 06:57:42.130533 12634 catalog_manager.cc:1347] Generated new cluster ID: ddb0763bf73141b6994903bbad2c07c4
I20260619 06:57:42.130602 12634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:42.136276 12634 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:42.137818 12634 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:42.147183 12634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0580eaeeecee43b8bb3a86fcb1b84d31: Generated new TSK 0
I20260619 06:57:42.148154 12634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:42.159617 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12552
2026-06-19T06:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:42 dist-test-slave-g8zg krb5kdc[12651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:42 dist-test-slave-g8zg krb5kdc[12651](info): set up 2 sockets
Jun 19 06:57:42 dist-test-slave-g8zg krb5kdc[12651](info): commencing operation
krb5kdc: starting...
W20260619 06:57:43.240212 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:43 dist-test-slave-g8zg krb5kdc[12651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:43.378666 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39471
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:39471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:43.488122 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:43.488437 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:43.488513 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:43.492481 12664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:43.492578 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:43.492614 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:43.492674 12664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:43.492714 12664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:43.497910 12664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:39471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:39471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:43.498970 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:43.500005 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:43.509407 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:43.509419 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:43.509419 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:43.509822 12664 server_base.cc:1034] running on GCE node
I20260619 06:57:43.510550 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:43.511940 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:43.513109 12664 hybrid_clock.cc:648] HybridClock initialized: now 1781852263513056 us; error 70 us; skew 500 ppm
Jun 19 06:57:43 dist-test-slave-g8zg krb5kdc[12651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852263, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:43.516350 12664 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:43.517316 12664 webserver.cc:460] Webserver started at http://127.22.113.62:34915/ using document root <none> and password file <none>
I20260619 06:57:43.517867 12664 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:43.517930 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:43.518105 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:43.520182 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "cb8577f46c914715acc88df4bee92dc7"
format_stamp: "Formatted at 2026-06-19 06:57:43 on dist-test-slave-g8zg"
server_key: "1ffcd151484e539382a3b5a0b178d581"
server_key_iv: "170b4209b1c77d9c6537b68f3cd78421"
server_key_version: "encryptionkey@0"
I20260619 06:57:43.520697 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "cb8577f46c914715acc88df4bee92dc7"
format_stamp: "Formatted at 2026-06-19 06:57:43 on dist-test-slave-g8zg"
server_key: "1ffcd151484e539382a3b5a0b178d581"
server_key_iv: "170b4209b1c77d9c6537b68f3cd78421"
server_key_version: "encryptionkey@0"
I20260619 06:57:43.524575 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 06:57:43.527220 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:43.528709 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:43.528971 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "cb8577f46c914715acc88df4bee92dc7"
format_stamp: "Formatted at 2026-06-19 06:57:43 on dist-test-slave-g8zg"
server_key: "1ffcd151484e539382a3b5a0b178d581"
server_key_iv: "170b4209b1c77d9c6537b68f3cd78421"
server_key_version: "encryptionkey@0"
I20260619 06:57:43.529093 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:43.533813 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:43.537153 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:43.537371 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:43.544798 12664 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:39471
I20260619 06:57:43.544826 12731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:39471 every 8 connection(s)
I20260619 06:57:43.545734 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:43.546042 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12664
I20260619 06:57:43.546211 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:43.546551 22980 external_mini_cluster.cc:1428] Setting key 35d6fb7b626479b9a8899f8a9b52ffab
I20260619 06:57:43.549647 12732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:43.552263 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:43.548213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51033 (local address 127.22.113.62:39471)
0619 06:57:43.548748 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:43.548765 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:43.548802 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:57:43.549499 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:43.549516 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:43.549867 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:43.550234 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:43.551068 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:43.551073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:43.551084 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:43.551094 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:43.551145 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:43.551164 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:43.551345 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:43.551420 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:57:43.551579 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":190,"threads_started":1}
I20260619 06:57:43.555754 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:43.557317 12732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: Bootstrap starting.
I20260619 06:57:43.559691 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:43.557926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44839 (local address 127.22.113.62:39471)
0619 06:57:43.558063 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:43.558068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:43.558280 (+ 212us) server_negotiation.cc:407] Connection header received
0619 06:57:43.558425 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:43.558428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:43.558463 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:43.558545 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:43.559296 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:43.559299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:43.559301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:57:43.559312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:43.559348 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:57:43.559358 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:43.559455 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:43.559530 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:57:43.559569 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:57:43.560400 12732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:43.561307 12732 log.cc:824] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:43.563529 12732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: No bootstrap required, opened a new log
I20260619 06:57:43.567410 12732 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } }
I20260619 06:57:43.567647 12732 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:43.567683 12732 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8577f46c914715acc88df4bee92dc7, State: Initialized, Role: FOLLOWER
I20260619 06:57:43.568081 12732 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } }
I20260619 06:57:43.568203 12732 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:43.568252 12732 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:43.568320 12732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:43.569219 12732 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } }
I20260619 06:57:43.569535 12732 leader_election.cc:302] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8577f46c914715acc88df4bee92dc7; no voters:
I20260619 06:57:43.569818 12732 leader_election.cc:288] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:43.570140 12742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:43.570628 12742 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [term 1 LEADER]: Becoming Leader. State: Replica: cb8577f46c914715acc88df4bee92dc7, State: Running, Role: LEADER
I20260619 06:57:43.571182 12742 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } }
I20260619 06:57:43.571369 12732 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:43.573351 12744 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8577f46c914715acc88df4bee92dc7. Latest consensus state: current_term: 1 leader_uuid: "cb8577f46c914715acc88df4bee92dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } } }
I20260619 06:57:43.573415 12743 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8577f46c914715acc88df4bee92dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8577f46c914715acc88df4bee92dc7" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 39471 } } }
I20260619 06:57:43.573508 12744 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:43.573544 12743 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7 [sys.catalog]: This master's current role is: LEADER
W20260619 06:57:43.577021 12757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:43.577116 12757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:43.577250 12756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:43.578100 12756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:43.588747 12756 catalog_manager.cc:1347] Generated new cluster ID: 235f0ac5ca46484e94e00e89bbc475b0
I20260619 06:57:43.588851 12756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:43.603288 12756 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:43.603760 12756 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:43.621551 12756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8577f46c914715acc88df4bee92dc7: Generated new TSK 0
I20260619 06:57:43.622493 12756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:57:43.627630 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12664
2026-06-19T06:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1468 ms)
[----------] 18 tests from AuthTokenIssuingTest (26214 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:43 dist-test-slave-g8zg krb5kdc[12763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:43 dist-test-slave-g8zg krb5kdc[12763](info): set up 2 sockets
Jun 19 06:57:43 dist-test-slave-g8zg krb5kdc[12763](info): commencing operation
krb5kdc: starting...
W20260619 06:57:44.688082 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:44 dist-test-slave-g8zg krb5kdc[12763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:44.822862 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35381
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:44.931783 12776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:44.932067 12776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:44.932091 12776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:44.932129 12776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:44.936146 12776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:44.936204 12776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:44.936224 12776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:44.936240 12776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:44.936256 12776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:44.941488 12776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:44.942446 12776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:44.943447 12776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:44.952301 12784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:44.952347 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:44.952301 12782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:44.953090 12776 server_base.cc:1034] running on GCE node
I20260619 06:57:44.953794 12776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:44.955181 12776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:44.956365 12776 hybrid_clock.cc:648] HybridClock initialized: now 1781852264956338 us; error 39 us; skew 500 ppm
Jun 19 06:57:44 dist-test-slave-g8zg krb5kdc[12763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:44.962926 12776 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:44.964129 12776 webserver.cc:460] Webserver started at http://127.22.113.62:33547/ using document root <none> and password file <none>
I20260619 06:57:44.964854 12776 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:44.964924 12776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:44.965166 12776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:44.967677 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "801e0465c051407c9481d30b19d29db1"
format_stamp: "Formatted at 2026-06-19 06:57:44 on dist-test-slave-g8zg"
server_key: "7d08289cf6d22757d110eff6ed3ed0fb"
server_key_iv: "a6a0276ef0b5900bb57e47c030edf24f"
server_key_version: "encryptionkey@0"
I20260619 06:57:44.968300 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "801e0465c051407c9481d30b19d29db1"
format_stamp: "Formatted at 2026-06-19 06:57:44 on dist-test-slave-g8zg"
server_key: "7d08289cf6d22757d110eff6ed3ed0fb"
server_key_iv: "a6a0276ef0b5900bb57e47c030edf24f"
server_key_version: "encryptionkey@0"
I20260619 06:57:44.972734 12776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 06:57:44.975634 12791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:44.977136 12776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:57:44.977295 12776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "801e0465c051407c9481d30b19d29db1"
format_stamp: "Formatted at 2026-06-19 06:57:44 on dist-test-slave-g8zg"
server_key: "7d08289cf6d22757d110eff6ed3ed0fb"
server_key_iv: "a6a0276ef0b5900bb57e47c030edf24f"
server_key_version: "encryptionkey@0"
I20260619 06:57:44.977416 12776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:45.008982 12776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:45.012984 12776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:45.013242 12776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:45.021736 12776 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:35381
I20260619 06:57:45.022087 12843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:35381 every 8 connection(s)
I20260619 06:57:45.022897 12776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:45.026185 12844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:45.030108 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12776
I20260619 06:57:45.030299 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:45.030650 22980 external_mini_cluster.cc:1428] Setting key 572202b6dcf80d7dfb3ac5dcc714fad1
I20260619 06:57:45.030892 12844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.61@KRBTEST.COM" created.
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:45.049868 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.034188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.22.113.62:35381)
0619 06:57:45.034746 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.034770 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.035100 (+ 330us) server_negotiation.cc:407] Connection header received
0619 06:57:45.036441 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.036447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.036637 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.036853 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.037850 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.038825 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.040082 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.040495 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.040702 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.044038 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.044050 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.044070 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.044108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.046050 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.046815 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.046820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.046822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.046936 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.047432 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.047436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.047439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.047829 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.047942 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.048552 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.048842 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":375,"thread_start_us":201,"threads_started":1}
I20260619 06:57:45.049872 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.034189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35381 (local address 127.0.0.1:58932)
0619 06:57:45.034917 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.034962 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.036087 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.036861 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.036867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.037396 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.037669 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.037684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.039002 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.039005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.039933 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.039943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.040742 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.040747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.040896 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.041677 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.041698 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.043804 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.046207 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.046221 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.046237 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.046622 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.047067 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.047073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.047078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.047241 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.047969 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.047984 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.048224 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.048472 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.048962 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":59,"spinlock_wait_cycles":34176,"thread_start_us":187,"threads_started":1}
I20260619 06:57:45.052901 12844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.053757 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.053457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51352)
0619 06:57:45.053591 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.053689 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:57:45.053977 12794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
I20260619 06:57:45.054926 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.054728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51368)
0619 06:57:45.054833 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.054878 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:57:45.055106 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:40763: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
I20260619 06:57:45.062896 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:40763
--webserver_interface=127.22.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:57:45.104494 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } attempt: 1
I20260619 06:57:45.105633 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.105283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51384)
0619 06:57:45.105460 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.105543 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:57:45.106424 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.106213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51396)
0619 06:57:45.106318 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.106373 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:57:45.106591 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:40763: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
W20260619 06:57:45.174520 12851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:45.174803 12851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:45.174829 12851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:45.174909 12851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:57:45.174827 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } attempt: 2
I20260619 06:57:45.175937 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.175596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51406)
0619 06:57:45.175769 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.175849 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 06:57:45.176704 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.176492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51420)
0619 06:57:45.176600 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.176646 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:57:45.176936 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:40763: Network error: Client connection negotiation failed: client connection to 127.22.113.61:40763: connect: Connection refused (error 111)
W20260619 06:57:45.179387 12851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:45.179461 12851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:45.179482 12851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:45.179499 12851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:45.179534 12851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:45.184881 12851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:45.185912 12851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:45.186896 12851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:45.195858 12857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:45.195876 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:45.195876 12859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:45.195947 12851 server_base.cc:1034] running on GCE node
I20260619 06:57:45.196931 12851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:45.198244 12851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:45.199433 12851 hybrid_clock.cc:648] HybridClock initialized: now 1781852265199408 us; error 40 us; skew 500 ppm
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:45.202613 12851 init.cc:377] Logged in from keytab as kudu/127.22.113.61@KRBTEST.COM (short username kudu)
I20260619 06:57:45.203770 12851 webserver.cc:460] Webserver started at http://127.22.113.61:38015/ using document root <none> and password file <none>
I20260619 06:57:45.204443 12851 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:45.204512 12851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:45.204777 12851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:45.207144 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data/instance:
uuid: "3096bdd3f383491ebddeb32b82b07093"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "447af32a00b2cb748ae39eab52abbd70"
server_key_iv: "eb3f3fb02cf9c2f45b4644c814ac6494"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.207834 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal/instance:
uuid: "3096bdd3f383491ebddeb32b82b07093"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "447af32a00b2cb748ae39eab52abbd70"
server_key_iv: "eb3f3fb02cf9c2f45b4644c814ac6494"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.212308 12851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:45.215692 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:45.217401 12851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:57:45.217587 12851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "3096bdd3f383491ebddeb32b82b07093"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "447af32a00b2cb748ae39eab52abbd70"
server_key_iv: "eb3f3fb02cf9c2f45b4644c814ac6494"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.217734 12851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:45.237550 12851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:45.241151 12851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:45.241389 12851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:45.249642 12851 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.61:40763
I20260619 06:57:45.249960 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.61:40763 every 8 connection(s)
I20260619 06:57:45.250870 12851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
I20260619 06:57:45.253875 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:45.257658 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.260344 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12851
I20260619 06:57:45.260495 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-1/wal/instance
I20260619 06:57:45.260756 22980 external_mini_cluster.cc:1428] Setting key 6e50d9002a98e15ea0c9b4817881975a
I20260619 06:57:45.268110 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } attempt: 3
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.60@KRBTEST.COM" created.
I20260619 06:57:45.274560 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.259797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.22.113.62:35381)
0619 06:57:45.259965 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.259970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.261048 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 06:57:45.262253 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.262259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.262334 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.262447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.264261 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.264787 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.266691 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.266961 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.267090 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.270142 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.270147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.270150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.270179 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.271930 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.272637 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.272661 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.272665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.272762 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.273172 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.273176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.273178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.273455 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.273565 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.274230 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.274366 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:57:45.275731 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.260255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35381 (local address 127.0.0.1:58934)
0619 06:57:45.260848 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.260905 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.261974 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.262694 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.262704 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.263275 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.264091 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.264115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.264954 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.264961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.266543 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.266553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.267131 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.267135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.267394 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.268122 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.268148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.269974 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.272048 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.272055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.272072 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.272493 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.272858 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.272860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.272862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.273068 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.273622 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.273631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.273958 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.274165 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.274517 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":177,"threads_started":1}
I20260619 06:57:45.278487 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.280995 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.268854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.22.113.61:40763)
0619 06:57:45.269382 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.269397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.269433 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:57:45.269506 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.269514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.269794 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.269997 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.270542 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.271770 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.272960 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.273576 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.273681 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.276249 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.276261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.276276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.276304 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.278351 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.278800 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.278806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.278808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.278905 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.279324 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.279329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.279331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.279749 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.279903 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.280434 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.280654 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":132,"threads_started":1}
I20260619 06:57:45.281612 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.268708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51436)
0619 06:57:45.268924 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.268941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.269030 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.270016 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.270023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.270249 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.270395 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.270404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.271916 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.271919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.272834 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.272845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.273708 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.273715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.273852 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.274361 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.274381 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.276092 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.278464 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.278470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.278472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.278680 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.279030 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.279035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.279038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.279155 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.279961 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.279964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.280093 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.280217 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.281417 (+ 1200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 06:57:45.284162 12844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.284988 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.284682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38568)
0619 06:57:45.284847 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.284912 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:57:45.285916 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.285639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38576)
0619 06:57:45.285813 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.285868 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
W20260619 06:57:45.286597 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
I20260619 06:57:45.290478 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.279132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51448)
0619 06:57:45.279353 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.279369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.279442 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.279799 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.279802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.279976 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.280056 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.280060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.280826 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.280830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.281605 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.281614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.282916 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.282920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.283019 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.283553 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.283566 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.286797 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.288907 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.288910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.288912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.289160 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.289469 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.289471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.289472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.289549 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.289951 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.289955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.290042 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.290149 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.290305 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 06:57:45.290521 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.279163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.22.113.61:40763)
0619 06:57:45.279592 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.279598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.279606 (+ 8us) server_negotiation.cc:407] Connection header received
0619 06:57:45.279627 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.279628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.279667 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.279792 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.280170 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.280714 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.282011 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.282797 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.283221 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.286953 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.286957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.286959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.286988 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.288778 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.289296 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.289302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.289304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.289398 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.289637 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.289639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.289640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.289850 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.289963 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.290203 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.290367 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":384,"thread_start_us":139,"threads_started":1}
I20260619 06:57:45.291556 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.292367 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.292060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38582)
0619 06:57:45.292227 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.292311 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 06:57:45.292695 12867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
I20260619 06:57:45.293782 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.293511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38590)
0619 06:57:45.293652 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.293708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:57:45.293954 12919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
I20260619 06:57:45.294502 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:42783
--webserver_interface=127.22.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:57:45.317832 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 1
I20260619 06:57:45.319025 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.318559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38604)
0619 06:57:45.318858 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.318936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 06:57:45.319839 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.319629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38608)
0619 06:57:45.319730 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.319784 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:57:45.319970 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
I20260619 06:57:45.343333 12919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 1
I20260619 06:57:45.344483 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.344096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38622)
0619 06:57:45.344309 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.344399 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 06:57:45.345422 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.345187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38628)
0619 06:57:45.345306 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.345357 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:57:45.345561 12919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
I20260619 06:57:45.395151 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 2
I20260619 06:57:45.396289 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.395920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38632)
0619 06:57:45.396124 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.396204 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 06:57:45.397141 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.396927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38644)
0619 06:57:45.397036 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.397093 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:57:45.397276 12844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
I20260619 06:57:45.413750 12919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 2
I20260619 06:57:45.414795 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.414423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38654)
0619 06:57:45.414633 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.414712 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 06:57:45.415604 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.415410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38670)
0619 06:57:45.415507 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.415561 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:57:45.415729 12919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:42783: Network error: Client connection negotiation failed: client connection to 127.22.113.60:42783: connect: Connection refused (error 111)
W20260619 06:57:45.417956 12927 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:45.418308 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:45.418370 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:45.418453 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:45.422667 12927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:45.422755 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:45.422802 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:45.422837 12927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:45.422868 12927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:45.428381 12927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35381,127.22.113.61:40763,127.22.113.60:42783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:45.429633 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:45.430895 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:45.440765 12935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:45.441113 12933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:45.441221 12932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:45.441540 12927 server_base.cc:1034] running on GCE node
I20260619 06:57:45.442169 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:45.443589 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:45.444793 12927 hybrid_clock.cc:648] HybridClock initialized: now 1781852265444772 us; error 33 us; skew 500 ppm
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:45.447886 12927 init.cc:377] Logged in from keytab as kudu/127.22.113.60@KRBTEST.COM (short username kudu)
I20260619 06:57:45.448915 12927 webserver.cc:460] Webserver started at http://127.22.113.60:46347/ using document root <none> and password file <none>
I20260619 06:57:45.449606 12927 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:45.449700 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:45.449954 12927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:45.452419 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data/instance:
uuid: "7236df40f3ff442985f5581e57a22b56"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "65f7b70e2f347bb714c137b42acb21fc"
server_key_iv: "0b93569b66f8a4ff735e8021ac9b9a63"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.453069 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal/instance:
uuid: "7236df40f3ff442985f5581e57a22b56"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "65f7b70e2f347bb714c137b42acb21fc"
server_key_iv: "0b93569b66f8a4ff735e8021ac9b9a63"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.457583 12927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:45.460578 12942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:45.462065 12927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:45.462247 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "7236df40f3ff442985f5581e57a22b56"
format_stamp: "Formatted at 2026-06-19 06:57:45 on dist-test-slave-g8zg"
server_key: "65f7b70e2f347bb714c137b42acb21fc"
server_key_iv: "0b93569b66f8a4ff735e8021ac9b9a63"
server_key_version: "encryptionkey@0"
I20260619 06:57:45.462375 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:45.477350 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:45.480682 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:45.480901 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:45.489423 12927 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.60:42783
I20260619 06:57:45.489796 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.60:42783 every 8 connection(s)
I20260619 06:57:45.490543 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
I20260619 06:57:45.491721 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 12927
I20260619 06:57:45.491873 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851976522130-22980-0/minicluster-data/master-2/wal/instance
I20260619 06:57:45.492206 22980 external_mini_cluster.cc:1428] Setting key 4fdd9d24051e519d3eeb1d9e00e10bd6
I20260619 06:57:45.494480 12995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:45.496460 12844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 3
I20260619 06:57:45.499930 12995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:45.506906 12919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } attempt: 3
I20260619 06:57:45.508052 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.497014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38680)
0619 06:57:45.497156 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.497204 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.497293 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.497770 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.497774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.497979 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.498075 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.498080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.498904 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.498910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.499520 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.499529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.500422 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.500428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.500548 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.501240 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.501256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.502921 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.505989 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.505994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.505997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.506221 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.506567 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.506572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.506574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.506663 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.507496 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.507499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.507594 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.507697 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.507857 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:57:45.510411 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.497244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.22.113.60:42783)
0619 06:57:45.497553 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.497561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.497571 (+ 10us) server_negotiation.cc:407] Connection header received
0619 06:57:45.497602 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.497604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.497636 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.497744 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.498197 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.498777 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.499640 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.500244 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.500407 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.503136 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.503142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.503154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.503181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.505877 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.506334 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.506340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.506346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.506470 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.506762 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.506766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.506768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.507365 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.508660 (+ 1295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.508878 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.509251 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":139,"threads_started":1}
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:45.510856 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.493815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.22.113.60:42783)
0619 06:57:45.494299 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.494317 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.494359 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:57:45.495160 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.495187 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.495624 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.495947 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.497033 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.498778 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.500122 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.500460 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.500563 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.503987 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.503992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.503997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.504023 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.506227 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.506947 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.506951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.506953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.507055 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.507455 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.507459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.507461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.507956 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.508152 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.509273 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.509377 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":257,"server-negotiator.queue_time_us":294,"thread_start_us":155,"threads_started":1}
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:45.515657 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.502312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35381 (local address 127.0.0.1:58948)
0619 06:57:45.502971 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.503028 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.503222 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.503714 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.503719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.504317 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.504500 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.504509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.505553 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.505558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.506695 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.506708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.507446 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.507449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.507617 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.508945 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.508958 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.511032 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.513585 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.513594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.513599 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.513942 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.514264 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.514267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.514269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.514351 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.514896 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.514904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.515037 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.515221 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.515441 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":7,"thread_start_us":121,"threads_started":1}
I20260619 06:57:45.515753 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.502415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.22.113.62:35381)
0619 06:57:45.502567 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.502574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.503111 (+ 537us) server_negotiation.cc:407] Connection header received
0619 06:57:45.503437 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.503442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.503502 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.503586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.504650 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.505418 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.506882 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.507185 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.507326 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.511236 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.511265 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.511268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.511301 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.513461 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.514057 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.514061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.514063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.514147 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.514450 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.514453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.514455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.514731 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.514804 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.515354 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.515522 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:57:45.517313 12995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.517860 12844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: Bootstrap starting.
I20260619 06:57:45.518711 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.507441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38692)
0619 06:57:45.507695 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.507711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.507795 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.508379 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.508382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.508579 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.508667 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.508673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.510245 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.510250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.511063 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.511076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.511781 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.511785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.511885 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.512468 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.512485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.514431 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.516581 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.516588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.516590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.516873 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.517234 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.517238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.517241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.517366 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.518113 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.518122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.518221 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.518371 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.518524 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 06:57:45.518783 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.507753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.22.113.60:42783)
0619 06:57:45.508119 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.508131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.508147 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:45.508186 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.508192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.508241 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.508358 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.508788 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.510127 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.511213 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.511648 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.511781 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.514591 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.514594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.514597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.514631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.516472 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.516991 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.516994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.516997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.517115 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.517493 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.517497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.517498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.517720 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.517824 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.518439 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.518608 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20260619 06:57:45.521256 12844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:45.522452 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: Bootstrap starting.
I20260619 06:57:45.522672 12844 log.cc:824] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:57:45.525383 12919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:45.525467 12844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: No bootstrap required, opened a new log
I20260619 06:57:45.526181 12919 log.cc:824] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:45.527956 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.517943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.22.113.61:40763)
0619 06:57:45.518060 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.518065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.518115 (+ 50us) server_negotiation.cc:407] Connection header received
0619 06:57:45.518276 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.518279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.518331 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.518381 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.519000 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.519835 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.520894 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.521268 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.521353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.524035 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.524039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.524042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.524065 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.526004 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.526483 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.526489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.526493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.526573 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.526886 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.526889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.526891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.527168 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.527268 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.527623 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.527773 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:57:45.528477 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.517896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:40763 (local address 127.0.0.1:51462)
0619 06:57:45.518040 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.518056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.518176 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.518522 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.518525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.518752 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.518869 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.518875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.519968 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.519971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.520762 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.520773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.521523 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.521528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.521637 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.522152 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.522165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.523874 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.526132 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.526135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.526137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.526370 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.526697 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.526700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.526702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.526785 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.527342 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.527345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.527445 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.528161 (+ 716us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.528317 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:57:45.528787 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: No bootstrap required, opened a new log
I20260619 06:57:45.529322 12995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:45.530362 12844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.530627 12844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e0465c051407c9481d30b19d29db1, State: Initialized, Role: FOLLOWER
I20260619 06:57:45.531046 12844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.532836 12844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:45.533702 12919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.533977 12919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096bdd3f383491ebddeb32b82b07093, State: Initialized, Role: FOLLOWER
I20260619 06:57:45.534348 12919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852265, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:45.535985 12919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:57:45.539857 13017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:45.539992 13017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:45.541023 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.529866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:42783 (local address 127.0.0.1:38706)
0619 06:57:45.530327 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:45.530342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:45.530454 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:45.530771 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:45.530774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:45.530944 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:45.531028 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.531034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.532071 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.532074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.532822 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:45.532832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.533517 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.533520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:45.533596 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.534115 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:45.534129 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:45.535900 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:45.539119 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.539124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.539126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.539481 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.539824 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:45.539827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:45.539829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:45.539924 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:45.540430 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:45.540433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:45.540524 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 06:57:45.540636 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:57:45.540810 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260619 06:57:45.541517 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.530200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.22.113.60:42783)
0619 06:57:45.530342 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.530348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.530396 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:57:45.530540 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.530542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.530581 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.530628 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.531144 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.531952 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.532947 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.533307 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.533382 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.536080 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.536085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.536090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.536114 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.538022 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.539608 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.539612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.539615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.539707 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.540028 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.540033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.540036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.540286 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.540392 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.541252 (+ 860us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.541422 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:57:45.542912 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.543105 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.543386 13011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.543553 13011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.544902 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.514024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.22.113.62:35381)
0619 06:57:45.515941 (+ 1917us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.515946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.526634 (+ 10688us) server_negotiation.cc:407] Connection header received
0619 06:57:45.526699 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.526702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.526767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.526884 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.527997 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.528952 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.530181 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.530504 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.530614 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.534392 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.534396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.534399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.534429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.542738 (+ 8309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.543446 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.543450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.543452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.543549 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.543989 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.543999 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.544001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.544187 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.544440 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.544605 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.544725 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":86,"threads_started":1}
I20260619 06:57:45.546027 12995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: Bootstrap starting.
I20260619 06:57:45.549314 12995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:45.550318 12995 log.cc:824] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: Log is configured to *not* fsync() on all Append() calls
W20260619 06:57:45.550750 13029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:45.550907 13029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:45.554268 12995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: No bootstrap required, opened a new log
I20260619 06:57:45.559182 12995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.559614 12995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7236df40f3ff442985f5581e57a22b56, State: Initialized, Role: FOLLOWER
I20260619 06:57:45.560024 12995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.561527 12995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:45.558269 12851 master_runner.cc:186] Error getting master registration for 127.22.113.60:42783: OK, instance_id { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" instance_seqno: 1781852265487086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:57:45.565258 12851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:57:45.568357 13043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:45.568450 13043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 06:57:45 dist-test-slave-g8zg krb5kdc[12763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:57:45.569012 13032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.569205 13032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.572495 12927 master_runner.cc:263] No leader master found from master 7236df40f3ff442985f5581e57a22b56
I20260619 06:57:45.572567 12927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:57:45.562739 12776 master_runner.cc:186] Error getting master registration for 127.22.113.60:42783: OK, instance_id { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" instance_seqno: 1781852265487086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:57:45.578790 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.553945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.22.113.61:40763)
0619 06:57:45.554070 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.554075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.554088 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:57:45.554127 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.554130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.554178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.554258 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.555221 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.556066 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.565430 (+ 9364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.565801 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.565908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.573278 (+ 7370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.573282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.573285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.573312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.575106 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.576888 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.576892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.576894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.576983 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.577286 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.577290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.577292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.577540 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.577796 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.578475 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.578601 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:57:45.579172 12776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:57:45.582234 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:45.583901 12818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "801e0465c051407c9481d30b19d29db1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58950
I20260619 06:57:45.584177 12818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:57:45.584296 12818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:45.585898 12818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.587703 12893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "801e0465c051407c9481d30b19d29db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3096bdd3f383491ebddeb32b82b07093"
I20260619 06:57:45.587976 12893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:45.589979 12893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801e0465c051407c9481d30b19d29db1 in term 1.
I20260619 06:57:45.590497 12794 leader_election.cc:302] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3096bdd3f383491ebddeb32b82b07093, 801e0465c051407c9481d30b19d29db1; no voters:
I20260619 06:57:45.590888 13005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:45.591343 13005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [term 1 LEADER]: Becoming Leader. State: Replica: 801e0465c051407c9481d30b19d29db1, State: Running, Role: LEADER
I20260619 06:57:45.591512 12969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "801e0465c051407c9481d30b19d29db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7236df40f3ff442985f5581e57a22b56"
I20260619 06:57:45.591777 12969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:45.591778 13005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } }
I20260619 06:57:45.593190 12969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801e0465c051407c9481d30b19d29db1 in term 1.
I20260619 06:57:45.593472 12818 leader_election.cc:288] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [CANDIDATE]: Term 1 election: Requested vote from peers 3096bdd3f383491ebddeb32b82b07093 (127.22.113.61:40763), 7236df40f3ff442985f5581e57a22b56 (127.22.113.60:42783)
I20260619 06:57:45.593922 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 801e0465c051407c9481d30b19d29db1. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.594062 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:45.603587 13047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:45.605368 13047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:45.616654 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.597466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.22.113.60:42783)
0619 06:57:45.597582 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.597586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.598405 (+ 819us) server_negotiation.cc:407] Connection header received
0619 06:57:45.598868 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.598871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.598924 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.598976 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.603093 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.603898 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.606253 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.606592 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.606665 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.609558 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.609563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.609566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.609591 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.611323 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.614207 (+ 2884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.614211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.614213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.614297 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.614947 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.614951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.614952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.615177 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.615324 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.616316 (+ 992us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.616448 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:57:45.619211 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.597103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.22.113.61:40763)
0619 06:57:45.597230 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.597235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.598589 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 06:57:45.600061 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.600065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.600117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.600167 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.601068 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.601885 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.605069 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.605404 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.605476 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.611841 (+ 6365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.611845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.611848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.611877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.613748 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.614586 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.614589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.614591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.614665 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.615636 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.615639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.615641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.615894 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.616000 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.618912 (+ 2912us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.619039 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:57:45.623914 12969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [term 1 FOLLOWER]: Refusing update from remote peer 801e0465c051407c9481d30b19d29db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:45.624749 13005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:45.627812 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:45.604134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.22.113.62:35381)
0619 06:57:45.604290 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.604295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.615634 (+ 11339us) server_negotiation.cc:407] Connection header received
0619 06:57:45.615864 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.615868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.615924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.615984 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.616999 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.617814 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.621625 (+ 3811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:45.621969 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:45.622055 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:45.624177 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:45.624181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:45.624187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:45.624219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:45.625924 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.626421 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.626425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.626427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.626506 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:45.626797 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:45.626801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:45.626802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:45.627025 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:45.627135 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:45.627503 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 06:57:45.627628 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:57:45.628906 13032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 801e0465c051407c9481d30b19d29db1. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.629055 13032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.631456 12893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [term 1 FOLLOWER]: Refusing update from remote peer 801e0465c051407c9481d30b19d29db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:45.632243 13005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:45.633550 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.633656 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:45.635346 13032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.635478 13032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.639078 13011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 801e0465c051407c9481d30b19d29db1. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.639211 13011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.643038 13055 mvcc.cc:205] Tried to move back new op lower bound from 7298466879983222784 to 7298466879868137472. Current Snapshot: MvccSnapshot[applied={T|T < 7298466879983222784}]
I20260619 06:57:45.643739 13056 mvcc.cc:205] Tried to move back new op lower bound from 7298466879983222784 to 7298466879868137472. Current Snapshot: MvccSnapshot[applied={T|T < 7298466879983222784}]
I20260619 06:57:45.651476 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.651612 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:45.652956 13011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "801e0465c051407c9481d30b19d29db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801e0465c051407c9481d30b19d29db1" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35381 } } peers { permanent_uuid: "3096bdd3f383491ebddeb32b82b07093" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 40763 } } peers { permanent_uuid: "7236df40f3ff442985f5581e57a22b56" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 42783 } } }
I20260619 06:57:45.653107 13011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:45.653968 13047 catalog_manager.cc:1347] Generated new cluster ID: 2d53437347724dadafc9047c7972557c
I20260619 06:57:45.654028 13047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:45.666114 13059 mvcc.cc:205] Tried to move back new op lower bound from 7298466879983222784 to 7298466879868137472. Current Snapshot: MvccSnapshot[applied={T|T < 7298466879983222784}]
I20260619 06:57:45.701862 13047 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:45.705758 13047 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:45.727128 13047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 801e0465c051407c9481d30b19d29db1: Generated new TSK 0
I20260619 06:57:45.728212 13047 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:57:45.762732 13001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.758223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.22.113.60:42783)
0619 06:57:45.758398 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.758404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.761012 (+ 2608us) server_negotiation.cc:407] Connection header received
0619 06:57:45.761622 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.761627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.761690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.761755 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.762572 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38724: BlockingRecv error: recv got EOF from 127.0.0.1:38724 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 06:57:45.764135 12847 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.758778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.22.113.62:35381)
0619 06:57:45.758910 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.758915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.760778 (+ 1863us) server_negotiation.cc:407] Connection header received
0619 06:57:45.763217 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.763222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.763334 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.763403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.764012 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58976: BlockingRecv error: recv got EOF from 127.0.0.1:58976 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 06:57:45.766577 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.759361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.22.113.61:40763)
0619 06:57:45.759505 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.759510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.761525 (+ 2015us) server_negotiation.cc:407] Connection header received
0619 06:57:45.764677 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.764681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.764742 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.764805 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.766445 (+ 1640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51484: BlockingRecv error: recv got EOF from 127.0.0.1:51484 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:57:45.773125 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.771420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.22.113.61:40763)
0619 06:57:45.771579 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.771585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.772049 (+ 464us) server_negotiation.cc:407] Connection header received
0619 06:57:45.772228 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.772231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.772338 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.772451 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.772956 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51494: BlockingRecv error: recv got EOF from 127.0.0.1:51494 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:57:45.777587 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.776590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.22.113.61:40763)
0619 06:57:45.776751 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.776756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.776772 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:45.776828 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.776831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.776888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.776987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.777474 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51506: BlockingRecv error: recv got EOF from 127.0.0.1:51506 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 06:57:45.789186 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.787729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.22.113.61:40763)
0619 06:57:45.787885 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.787890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.788154 (+ 264us) server_negotiation.cc:407] Connection header received
0619 06:57:45.788316 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.788319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.788388 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.788518 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.789034 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51522: BlockingRecv error: recv got EOF from 127.0.0.1:51522 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:57:45.797016 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.795462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.22.113.61:40763)
0619 06:57:45.795645 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.795650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.795984 (+ 334us) server_negotiation.cc:407] Connection header received
0619 06:57:45.796143 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.796147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.796243 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.796366 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.796892 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51528: BlockingRecv error: recv got EOF from 127.0.0.1:51528 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 06:57:45.808144 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.806953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.22.113.61:40763)
0619 06:57:45.807094 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.807098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.807309 (+ 211us) server_negotiation.cc:407] Connection header received
0619 06:57:45.807451 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.807453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.807507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.807598 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.807988 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51534: BlockingRecv error: recv got EOF from 127.0.0.1:51534 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 06:57:45.827979 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.826609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.22.113.61:40763)
0619 06:57:45.826806 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.826811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.826949 (+ 138us) server_negotiation.cc:407] Connection header received
0619 06:57:45.827143 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.827149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.827204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.827340 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.827866 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51548: BlockingRecv error: recv got EOF from 127.0.0.1:51548 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 06:57:45.864120 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.862597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.22.113.61:40763)
0619 06:57:45.862792 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.862799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.863043 (+ 244us) server_negotiation.cc:407] Connection header received
0619 06:57:45.863292 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.863295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.863362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.863463 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.863979 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51556: BlockingRecv error: recv got EOF from 127.0.0.1:51556 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 06:57:45.932535 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:45.930871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.22.113.61:40763)
0619 06:57:45.931112 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:45.931118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:45.931297 (+ 179us) server_negotiation.cc:407] Connection header received
0619 06:57:45.931549 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:45.931558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:45.931616 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:45.931727 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:45.932296 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51568: BlockingRecv error: recv got EOF from 127.0.0.1:51568 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 06:57:46.064796 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:46.062893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.22.113.61:40763)
0619 06:57:46.063129 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.063133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.063603 (+ 470us) server_negotiation.cc:407] Connection header received
0619 06:57:46.063801 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.063805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.063992 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.064113 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:46.064632 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51584: BlockingRecv error: recv got EOF from 127.0.0.1:51584 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260619 06:57:46.324389 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:46.323151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.22.113.61:40763)
0619 06:57:46.323340 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.323344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.323498 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:57:46.323563 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.323565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.323619 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.323704 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:46.324185 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51592: BlockingRecv error: recv got EOF from 127.0.0.1:51592 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:57:46.552323 13029 catalog_manager.cc:1259] Loaded cluster ID: 2d53437347724dadafc9047c7972557c
I20260619 06:57:46.552400 13029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: loading cluster ID for follower catalog manager: success
I20260619 06:57:46.554020 13029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: acquiring CA information for follower catalog manager: success
I20260619 06:57:46.555094 13029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3096bdd3f383491ebddeb32b82b07093: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:57:46.569574 13043 catalog_manager.cc:1259] Loaded cluster ID: 2d53437347724dadafc9047c7972557c
I20260619 06:57:46.569643 13043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: loading cluster ID for follower catalog manager: success
I20260619 06:57:46.571175 13043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: acquiring CA information for follower catalog manager: success
I20260619 06:57:46.572230 13043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7236df40f3ff442985f5581e57a22b56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:57:46.842522 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:46.838610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.22.113.61:40763)
0619 06:57:46.838858 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.838863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.839001 (+ 138us) server_negotiation.cc:407] Connection header received
0619 06:57:46.839115 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.839117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.839333 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.839466 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:46.839686 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.840212 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.841073 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.841286 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.841430 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:46.841692 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:46.842118 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:46.842209 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:46.842277 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 06:57:46.842324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260619 06:57:46.849583 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:46.845658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.22.113.60:42783)
0619 06:57:46.845938 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.845945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.846011 (+ 66us) server_negotiation.cc:407] Connection header received
0619 06:57:46.846208 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.846211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.846335 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.846506 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:46.846726 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.847273 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.848104 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.848330 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.848445 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:46.848757 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:46.849169 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:46.849302 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:46.849362 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 06:57:46.849424 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":67,"threads_started":1}
I20260619 06:57:46.856643 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:46.852744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.22.113.61:40763)
0619 06:57:46.852886 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.852891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.853205 (+ 314us) server_negotiation.cc:407] Connection header received
0619 06:57:46.853392 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.853395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.853449 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.853523 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:46.853757 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.854518 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.855552 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.855865 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.855955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:46.856184 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:46.856266 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:46.856311 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:46.856459 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 06:57:46.856507 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:57:46.857131 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:46.852788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.22.113.60:42783)
0619 06:57:46.852933 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.852939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.853227 (+ 288us) server_negotiation.cc:407] Connection header received
0619 06:57:46.853637 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.853642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.853693 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.853754 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:46.854085 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.854829 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.855899 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.856226 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.856335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:46.856692 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:46.856758 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:46.856803 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:46.856969 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 06:57:46.857014 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:57:46.857442 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:46.853051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.22.113.62:35381)
0619 06:57:46.853289 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:46.853293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:46.853303 (+ 10us) server_negotiation.cc:407] Connection header received
0619 06:57:46.853419 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:46.853422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:46.853563 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:46.853694 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:46.853912 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.854492 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.855670 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:46.856025 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:46.856139 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:46.856482 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:46.856819 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:46.856905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:46.857238 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:57:46.857302 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260619 06:57:46.860322 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12776
I20260619 06:57:46.867357 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12851
I20260619 06:57:46.873087 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 12927
2026-06-19T06:57:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3245 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:46 dist-test-slave-g8zg krb5kdc[13145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:46 dist-test-slave-g8zg krb5kdc[13145](info): set up 2 sockets
Jun 19 06:57:46 dist-test-slave-g8zg krb5kdc[13145](info): commencing operation
krb5kdc: starting...
W20260619 06:57:47.923890 22980 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/test-user.keytab.
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
Entry for principal kudu/127.22.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab.
I20260619 06:57:48.058527 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35419
--webserver_interface=127.22.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:48.172314 13158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:48.172622 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:48.172652 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:48.172694 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:48.176728 13158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:48.176802 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:48.176831 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:48.176851 13158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:48.176869 13158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:48.182065 13158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.62.keytab
--principal=kudu/127.22.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:48.183096 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:48.184172 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:48.193325 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.193392 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.193328 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:48.193578 13158 server_base.cc:1034] running on GCE node
I20260619 06:57:48.194454 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:48.195900 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:48.197091 13158 hybrid_clock.cc:648] HybridClock initialized: now 1781852268197070 us; error 38 us; skew 500 ppm
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:48.200155 13158 init.cc:377] Logged in from keytab as kudu/127.22.113.62@KRBTEST.COM (short username kudu)
I20260619 06:57:48.201123 13158 webserver.cc:460] Webserver started at http://127.22.113.62:35991/ using document root <none> and password file <none>
I20260619 06:57:48.201732 13158 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:48.201792 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:48.201983 13158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:48.204300 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data/instance:
uuid: "be873e30984b42c987503cabb8a711ca"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "2be5a5ea30b0405b79f4d6bb2ab72207"
server_key_iv: "70844debc2d2254376df2f625ecb5571"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.204869 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance:
uuid: "be873e30984b42c987503cabb8a711ca"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "2be5a5ea30b0405b79f4d6bb2ab72207"
server_key_iv: "70844debc2d2254376df2f625ecb5571"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.209075 13158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:48.211719 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:48.213132 13158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:48.213239 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal
uuid: "be873e30984b42c987503cabb8a711ca"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "2be5a5ea30b0405b79f4d6bb2ab72207"
server_key_iv: "70844debc2d2254376df2f625ecb5571"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.213330 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:48.238372 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:48.241636 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:48.241819 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:48.250206 13158 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.62:35419
I20260619 06:57:48.250535 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.62:35419 every 8 connection(s)
I20260619 06:57:48.251410 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:48.254438 13226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:48.255815 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 13158
I20260619 06:57:48.255967 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-0/wal/instance
I20260619 06:57:48.256376 22980 external_mini_cluster.cc:1428] Setting key 01cf8fc01a9a6a7153defc91009d082d
I20260619 06:57:48.258805 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.61@KRBTEST.COM" created.
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:48.277771 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.262154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35419 (local address 127.0.0.1:37632)
0619 06:57:48.262773 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.262833 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.263950 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.264918 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.264923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.265306 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.265692 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.265708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.266989 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.266992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.268100 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.268110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.268810 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.268814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.268930 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.269724 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.269752 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.271670 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.274410 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.274418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.274427 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.274674 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.275098 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.275101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.275102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.275208 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.276135 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.276141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.276286 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.276437 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.276729 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":175,"threads_started":1}
I20260619 06:57:48.277773 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.262154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.22.113.62:35419)
0619 06:57:48.262901 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.262915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.262983 (+ 68us) server_negotiation.cc:407] Connection header received
0619 06:57:48.264330 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.264339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.264650 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.264931 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.265890 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.266854 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.268258 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.268642 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.268822 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.271897 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.271907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.271925 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.271968 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.274273 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.274818 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.274828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.274830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.274997 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.275375 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.275379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.275380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.276003 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.276161 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.276603 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.276807 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"spinlock_wait_cycles":18304,"thread_start_us":191,"threads_started":1}
I20260619 06:57:48.280752 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.281600 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.281300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45354)
0619 06:57:48.281428 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.281539 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:57:48.281867 13174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
I20260619 06:57:48.282953 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.282756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45360)
0619 06:57:48.282861 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.282904 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:57:48.283239 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33669: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
Entry for principal kudu/127.22.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab.
I20260619 06:57:48.287039 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:33669
--webserver_interface=127.22.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:57:48.332556 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } attempt: 1
I20260619 06:57:48.333706 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.333318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45366)
0619 06:57:48.333524 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.333609 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:57:48.334499 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.334290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45376)
0619 06:57:48.334392 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.334448 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:57:48.334635 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33669: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
W20260619 06:57:48.399880 13233 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:48.400206 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:48.400274 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:48.400358 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:57:48.402819 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } attempt: 2
I20260619 06:57:48.403882 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.403526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45388)
0619 06:57:48.403721 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.403795 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 06:57:48.404577 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.404374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45390)
0619 06:57:48.404475 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.404526 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:57:48.404596 13233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:48.404676 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:48.404711 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.61:33669: Network error: Client connection negotiation failed: client connection to 127.22.113.61:33669: connect: Connection refused (error 111)
W20260619 06:57:48.404736 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:48.404794 13233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:48.404824 13233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:48.410029 13233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.61.keytab
--principal=kudu/127.22.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.61:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.113.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:48.411136 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:48.412201 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:48.421398 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.421412 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.421412 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:48.422008 13233 server_base.cc:1034] running on GCE node
I20260619 06:57:48.422719 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:48.424105 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:48.425302 13233 hybrid_clock.cc:648] HybridClock initialized: now 1781852268425299 us; error 48 us; skew 500 ppm
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:48.428473 13233 init.cc:377] Logged in from keytab as kudu/127.22.113.61@KRBTEST.COM (short username kudu)
I20260619 06:57:48.429486 13233 webserver.cc:460] Webserver started at http://127.22.113.61:34333/ using document root <none> and password file <none>
I20260619 06:57:48.430157 13233 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:48.430255 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:48.430519 13233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:48.432907 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data/instance:
uuid: "2d0e12ac6ced48179537db91176babbb"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "aecf9d5b973c44e9f1f81bd4f967302d"
server_key_iv: "ac630dceb358df564952c60c3b8a70e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.433537 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal/instance:
uuid: "2d0e12ac6ced48179537db91176babbb"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "aecf9d5b973c44e9f1f81bd4f967302d"
server_key_iv: "ac630dceb358df564952c60c3b8a70e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.437805 13233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:48.440655 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:48.442207 13233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:48.442363 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal
uuid: "2d0e12ac6ced48179537db91176babbb"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "aecf9d5b973c44e9f1f81bd4f967302d"
server_key_iv: "ac630dceb358df564952c60c3b8a70e3"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.442483 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:48.471431 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:48.475148 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:48.475407 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:48.483845 13233 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.61:33669
I20260619 06:57:48.484213 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.61:33669 every 8 connection(s)
I20260619 06:57:48.485003 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/data/info.pb
I20260619 06:57:48.488031 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:48.491782 13301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.494599 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 13233
I20260619 06:57:48.494728 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-1/wal/instance
I20260619 06:57:48.495075 22980 external_mini_cluster.cc:1428] Setting key 84e5b771bd166ec3dbd231fed34d1a07
I20260619 06:57:48.495891 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } attempt: 3
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.113.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.113.60@KRBTEST.COM" created.
I20260619 06:57:48.508492 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.496540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.22.113.61:33669)
0619 06:57:48.497006 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.497021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.497051 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:57:48.497113 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.497121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.497363 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.497557 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.498050 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.498965 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500312 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.500853 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500990 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.503383 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.503392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.503408 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.503441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.505224 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.505726 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.505734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.505742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.505887 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.506170 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.506173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.506176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.506636 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.506802 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.507318 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.507669 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":293,"spinlock_wait_cycles":1280,"thread_start_us":172,"threads_started":1}
I20260619 06:57:48.509383 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.496358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45394)
0619 06:57:48.496557 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.496575 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.496686 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.497569 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.497573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.497783 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.497904 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.497915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.499171 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.499178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.500177 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.500190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.500975 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.501088 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.501725 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.501743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.503182 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.505389 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.505393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.505395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.505605 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.505996 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.505998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.505999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.506074 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.506782 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.506784 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.506949 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.507074 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.507303 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":3}
I20260619 06:57:48.510330 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.494037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35419 (local address 127.0.0.1:37638)
0619 06:57:48.494586 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.494643 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.495888 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.496674 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.496688 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.497147 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.497831 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.497852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.499070 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.499076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.500009 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.500019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.500696 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.500996 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.501992 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.502019 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.503979 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.506314 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.506326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.506336 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.506615 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.507370 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.507374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.507376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.507502 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.509602 (+ 2100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.509609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.509730 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.509928 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.510134 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":102,"spinlock_wait_cycles":2944,"thread_start_us":131,"threads_started":1}
I20260619 06:57:48.511183 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.493733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.22.113.62:35419)
0619 06:57:48.493934 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.493940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.494813 (+ 873us) server_negotiation.cc:407] Connection header received
0619 06:57:48.496170 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.496175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.496244 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.496339 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.498020 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.498921 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500132 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.500490 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.500603 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.504143 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.504147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.504149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.504168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.506195 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.506740 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.506745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.506746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.506840 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.507667 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.507670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.507672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.509464 (+ 1792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.509605 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.510961 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.511087 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:57:48.511962 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.512514 13301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.512723 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.512498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40754)
0619 06:57:48.512614 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.512669 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:57:48.513659 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.513381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40766)
0619 06:57:48.513548 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.513604 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 06:57:48.513907 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:57:48.523962 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.513337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45396)
0619 06:57:48.513452 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.513762 (+ 310us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.513874 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.514169 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.514171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.514320 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.514397 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.514402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.515145 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.515151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.515763 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.515772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.516250 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.516253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.516382 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.516912 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.516926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.520453 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.522257 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.522261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.522263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.522496 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.522766 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.522771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.522773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.522856 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.523303 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.523306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.523401 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.523567 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.523736 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:57:48.523962 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.513483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.22.113.61:33669)
0619 06:57:48.513577 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.513583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.513949 (+ 366us) server_negotiation.cc:407] Connection header received
0619 06:57:48.513971 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.513972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.514002 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.514077 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.514503 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.515040 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.515879 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.516136 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.516247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.520592 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.520596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.520599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.520629 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.522131 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.522595 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.522600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.522602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.522676 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.522955 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.522961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.522963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.523165 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.523315 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.523627 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.523795 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":38}
I20260619 06:57:48.525058 13301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
Entry for principal kudu/127.22.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab.
I20260619 06:57:48.525864 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.525603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40772)
0619 06:57:48.525700 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.525808 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 06:57:48.526568 22980 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9tRik3/build/debug/bin/kudu
/tmp/dist-test-task9tRik3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:38793
--webserver_interface=127.22.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.113.20:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:48.526615 13251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
I20260619 06:57:48.527662 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.527408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40780)
0619 06:57:48.527534 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.527586 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:57:48.527872 13301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
I20260619 06:57:48.545104 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 1
I20260619 06:57:48.546118 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.545641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40786)
0619 06:57:48.545851 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.545959 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:57:48.546916 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.546713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40796)
0619 06:57:48.546815 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.546870 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:57:48.547072 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
I20260619 06:57:48.577208 13301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 1
I20260619 06:57:48.578337 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.577941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40800)
0619 06:57:48.578137 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.578236 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:57:48.579139 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.578929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40806)
0619 06:57:48.579030 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.579085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:57:48.579275 13301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
I20260619 06:57:48.622268 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 2
I20260619 06:57:48.623349 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.622940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40818)
0619 06:57:48.623127 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.623203 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:57:48.624132 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.623930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40828)
0619 06:57:48.624028 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.624075 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:57:48.624264 13226 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
W20260619 06:57:48.644626 13308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:57:48.644970 13308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:48.645040 13308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:48.645131 13308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:57:48.647428 13301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 2
I20260619 06:57:48.648353 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.648030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40840)
0619 06:57:48.648189 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.648265 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:57:48.649051 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.648841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40844)
0619 06:57:48.648941 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.648992 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:57:48.649181 13301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.113.60:38793: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
W20260619 06:57:48.649617 13308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:48.649698 13308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:48.649746 13308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:48.649778 13308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:48.649813 13308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:48.655170 13308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.113.20:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.113.62:35419,127.22.113.61:33669,127.22.113.60:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/krb5kdc/kudu_127.22.113.60.keytab
--principal=kudu/127.22.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.113.60:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.113.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:48.656330 13308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:48.657404 13308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:48.666525 13314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.666525 13313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:48.666538 13316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:48.666906 13308 server_base.cc:1034] running on GCE node
I20260619 06:57:48.667697 13308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:48.669001 13308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:48.670172 13308 hybrid_clock.cc:648] HybridClock initialized: now 1781852268670150 us; error 39 us; skew 500 ppm
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:48.673442 13308 init.cc:377] Logged in from keytab as kudu/127.22.113.60@KRBTEST.COM (short username kudu)
I20260619 06:57:48.674424 13308 webserver.cc:460] Webserver started at http://127.22.113.60:39869/ using document root <none> and password file <none>
I20260619 06:57:48.675100 13308 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:48.675194 13308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:48.675467 13308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:48.677770 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data/instance:
uuid: "a0e249cf962143e49f8870da973822c3"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "5b239678a42e286b168332b4c87be8d1"
server_key_iv: "fae52cdaeb88ebb0155b17a1c66759f8"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.678388 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal/instance:
uuid: "a0e249cf962143e49f8870da973822c3"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "5b239678a42e286b168332b4c87be8d1"
server_key_iv: "fae52cdaeb88ebb0155b17a1c66759f8"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.683741 13308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:57:48.686479 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:48.687904 13308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:48.688050 13308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data,/tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal
uuid: "a0e249cf962143e49f8870da973822c3"
format_stamp: "Formatted at 2026-06-19 06:57:48 on dist-test-slave-g8zg"
server_key: "5b239678a42e286b168332b4c87be8d1"
server_key_iv: "fae52cdaeb88ebb0155b17a1c66759f8"
server_key_version: "encryptionkey@0"
I20260619 06:57:48.688167 13308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:48.712517 13308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:48.715708 13308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:48.715931 13308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:48.723435 13226 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 3
I20260619 06:57:48.724359 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.724029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40854)
0619 06:57:48.724185 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.724268 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.113.60:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:57:48.724956 13308 rpc_server.cc:307] RPC server started. Bound to: 127.22.113.60:38793
I20260619 06:57:48.725353 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.113.60:38793 every 8 connection(s)
I20260619 06:57:48.726173 13308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/data/info.pb
I20260619 06:57:48.729529 13376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:48.733691 22980 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9tRik3/build/debug/bin/kudu as pid 13308
I20260619 06:57:48.733841 22980 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9tRik3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851976522130-22980-0/minicluster-data/master-2/wal/instance
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.62@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:48.734201 22980 external_mini_cluster.cc:1428] Setting key 7109bc528e0402413ca9189ee251c2fb
I20260619 06:57:48.734519 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.739564 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.725026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40868)
0619 06:57:48.725188 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.725207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.725323 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.728605 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.728608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.728817 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.728940 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.728947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.730359 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.730362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.731158 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.731169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.732107 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.732112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.732197 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.732753 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.732769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.734470 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.737388 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.737392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.737395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.737682 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.738088 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.738095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.738100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.738196 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.738989 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.738992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.739100 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.739217 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.739396 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:57:48.740371 13301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } attempt: 3
I20260619 06:57:48.741313 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.726521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.22.113.60:38793)
0619 06:57:48.726986 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.727014 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.727056 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:57:48.727876 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.727894 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.728279 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.728609 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.729079 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.730220 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.731378 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.731976 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.732107 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.734733 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.734744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.734763 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.734801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.737255 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.737832 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.737840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.737847 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.737979 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.738315 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.738318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.738319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.738864 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.739001 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.739417 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.739749 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":138,"threads_started":1}
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.61@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:48.748231 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: Bootstrap starting.
I20260619 06:57:48.748431 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.734934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.22.113.60:38793)
0619 06:57:48.735354 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.735366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.735721 (+ 355us) server_negotiation.cc:407] Connection header received
0619 06:57:48.735861 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.735865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.735919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.736020 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.737231 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.738042 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.739197 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.739537 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.739632 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.742648 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.742652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.742656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.742685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.744388 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.745315 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.745322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.745327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.745427 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.745850 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.745856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.745857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.746117 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.746317 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.748074 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.748222 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"spinlock_wait_cycles":824704,"thread_start_us":175,"threads_started":1}
I20260619 06:57:48.750454 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.736542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.22.113.62:35419)
0619 06:57:48.736685 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.736689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.737049 (+ 360us) server_negotiation.cc:407] Connection header received
0619 06:57:48.737265 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.737268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.737324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.737405 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.738118 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.738635 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.740017 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.740368 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.740441 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.745326 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.745330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.745333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.745360 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.747120 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.748083 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.748089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.748091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.748191 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.749058 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.749061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.749064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.749553 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.749650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.750120 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.750260 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:57:48.751372 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.736482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.62:35419 (local address 127.0.0.1:37644)
0619 06:57:48.736931 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.736979 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.737142 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.737440 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.737446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.737805 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.737965 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.737975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.738755 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.738759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.739900 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.739911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.740595 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.740601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.740715 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.741320 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.741335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.745167 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.747297 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.747307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.747312 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.747962 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.748587 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.748592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.748595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.748684 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.749720 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.749726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.749854 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.750937 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.751134 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":100,"threads_started":1}
I20260619 06:57:48.751725 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.741148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40886)
0619 06:57:48.741365 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.741379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.741461 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.741885 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.741888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.742067 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.742173 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.742179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.743984 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.743988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.744794 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.744805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.745409 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.745413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.745517 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.746482 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.746501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.747808 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.750169 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.750172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.750174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.750397 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.750705 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.750709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.750711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.750806 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.751221 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.751224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.751346 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.751434 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.751581 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 06:57:48.752368 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.741095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.22.113.60:38793)
0619 06:57:48.741653 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.741662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.741674 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:57:48.741704 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.741707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.741747 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.742275 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.742370 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.743850 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.744933 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.745273 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.746413 (+ 1140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.747939 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.747943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.747946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.747967 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.750060 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.750520 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.750523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.750524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.750599 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.750921 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.750927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.750930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.751111 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.752022 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.752096 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.752218 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":84,"threads_started":1}
I20260619 06:57:48.752416 13226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:48.753149 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.753497 13226 log.cc:824] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:48.755759 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: Bootstrap starting.
I20260619 06:57:48.756067 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: No bootstrap required, opened a new log
I20260619 06:57:48.758380 13301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:48.759235 13301 log.cc:824] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
I20260619 06:57:48.760663 13226 raft_consensus.cc:348] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.761003 13226 raft_consensus.cc:729] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be873e30984b42c987503cabb8a711ca, State: Initialized, Role: FOLLOWER
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.62@KRBTEST.COM
I20260619 06:57:48.761428 13226 consensus_queue.cc:260] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.762050 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.762954 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.763106 13226 sys_catalog.cc:564] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:48.764240 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: No bootstrap required, opened a new log
I20260619 06:57:48.764779 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.753609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.61:33669 (local address 127.0.0.1:45410)
0619 06:57:48.754206 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.754220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.754304 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.754613 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.754616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.754787 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.754896 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.754902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.756303 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.756307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.757119 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.757130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.757774 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.757778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.757877 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.758909 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.758924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.760512 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.762479 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.762482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.762484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.762740 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.763411 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.763414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.763416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.763513 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.764115 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.764118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.764243 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.764365 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.764551 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":109,"threads_started":1}
I20260619 06:57:48.765831 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.753924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.22.113.61:33669)
0619 06:57:48.754353 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.754359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.754375 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:48.754424 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.754427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.754483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.754577 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.755467 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.756161 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.757263 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.757629 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.757742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.760654 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.760659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.760662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.760688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.762358 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.762879 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.762884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.762886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.763286 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.763620 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.763623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.763626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.763949 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.764062 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.765473 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.765619 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260619 06:57:48.765942 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.750258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.22.113.62:35419)
0619 06:57:48.750671 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.750677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.750691 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:57:48.750723 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.750726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.750771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.750857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.751630 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.752370 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.753762 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.754077 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.754159 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.762005 (+ 7846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.762009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.762012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.762038 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.763817 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.764421 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.764424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.764426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.764512 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.764834 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.764837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.764839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.765077 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.765255 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.765646 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.765770 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":103,"threads_started":1}
I20260619 06:57:48.767544 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:57:48.770772 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.771104 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0e12ac6ced48179537db91176babbb, State: Initialized, Role: FOLLOWER
I20260619 06:57:48.771517 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.773036 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:48.769685 13158 master_runner.cc:186] Error getting master registration for 127.22.113.61:33669: OK, instance_id { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" instance_seqno: 1781852268481389 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:57:48.773180 13158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.113.61@KRBTEST.COM
W20260619 06:57:48.769692 13398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:48.777648 13398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 06:57:48 dist-test-slave-g8zg krb5kdc[13145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852268, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.113.60@KRBTEST.COM for kudu/127.22.113.60@KRBTEST.COM
I20260619 06:57:48.781028 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.767688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.22.113.61:33669)
0619 06:57:48.767797 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.767802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.768023 (+ 221us) server_negotiation.cc:407] Connection header received
0619 06:57:48.768137 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.768140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.768191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.768269 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.769156 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.769891 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.770965 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.771442 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.771521 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.774877 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.774882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.774885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.774915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.777299 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.779099 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.779106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.779110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.779204 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.779563 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.779567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.779569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.780095 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.780299 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.780685 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.780840 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 06:57:48.782444 13411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:48.782526 13411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:48.780174 13233 master_runner.cc:186] Error getting master registration for 127.22.113.60:38793: OK, instance_id { permanent_uuid: "a0e249cf962143e49f8870da973822c3" instance_seqno: 1781852268722507 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:57:48.783200 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.784309 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.783701 13233 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:57:48.785974 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.768091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.113.60:38793 (local address 127.0.0.1:40894)
0619 06:57:48.768667 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:48.768681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:48.768762 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:48.773535 (+ 4773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:48.773538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:48.773741 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:48.773849 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.773858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.775011 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.775014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.775814 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:48.775826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.776366 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.776369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:48.776453 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.776954 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:48.776969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:48.778700 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:48.784118 (+ 5418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.784122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.784125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.784411 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.784958 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:48.784961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:48.784964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:48.785058 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:48.785488 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:48.785490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:48.785563 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 06:57:48.785637 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:57:48.785797 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511}
I20260619 06:57:48.786490 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.768328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.22.113.60:38793)
0619 06:57:48.768433 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.768437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.773308 (+ 4871us) server_negotiation.cc:407] Connection header received
0619 06:57:48.773354 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.773357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.773409 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.773959 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.774049 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.774890 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.775940 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.776234 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.776692 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.779450 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.779454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.779456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.779484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.783986 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.784514 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.784517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.784519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.784609 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.785163 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.785166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.785168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.785389 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.786161 (+ 772us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.786234 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.786353 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:57:48.787465 22980 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:57:48.788904 13200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "be873e30984b42c987503cabb8a711ca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37656
I20260619 06:57:48.789209 13200 raft_consensus.cc:482] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:57:48.789331 13200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:48.789512 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: Bootstrap starting.
I20260619 06:57:48.790903 13200 raft_consensus.cc:504] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.792654 13376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:48.792830 13275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be873e30984b42c987503cabb8a711ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d0e12ac6ced48179537db91176babbb"
I20260619 06:57:48.793130 13275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:48.793632 13376 log.cc:824] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:48.794703 13275 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be873e30984b42c987503cabb8a711ca in term 1.
I20260619 06:57:48.795177 13174 leader_election.cc:302] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0e12ac6ced48179537db91176babbb, be873e30984b42c987503cabb8a711ca; no voters:
I20260619 06:57:48.795542 13387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:48.795476 13200 leader_election.cc:288] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [CANDIDATE]: Term 1 election: Requested vote from peers 2d0e12ac6ced48179537db91176babbb (127.22.113.61:33669), a0e249cf962143e49f8870da973822c3 (127.22.113.60:38793)
I20260619 06:57:48.795791 13387 raft_consensus.cc:686] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [term 1 LEADER]: Becoming Leader. State: Replica: be873e30984b42c987503cabb8a711ca, State: Running, Role: LEADER
I20260619 06:57:48.796156 13387 consensus_queue.cc:237] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.798027 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader be873e30984b42c987503cabb8a711ca. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.798126 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:48.801838 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: No bootstrap required, opened a new log
I20260619 06:57:48.806510 13376 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.806770 13376 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e249cf962143e49f8870da973822c3, State: Initialized, Role: FOLLOWER
I20260619 06:57:48.807113 13420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:48.807123 13376 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } }
I20260619 06:57:48.807838 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.808136 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.808404 13420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:48.808503 13376 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:48.808547 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be873e30984b42c987503cabb8a711ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0e249cf962143e49f8870da973822c3"
I20260619 06:57:48.808830 13350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:48.810407 13350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be873e30984b42c987503cabb8a711ca in term 1.
I20260619 06:57:48.814534 13275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [term 1 FOLLOWER]: Refusing update from remote peer be873e30984b42c987503cabb8a711ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:48.814533 13350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [term 1 FOLLOWER]: Refusing update from remote peer be873e30984b42c987503cabb8a711ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:57:48.815191 13387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:48.815743 13387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:57:48.821507 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.797824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.22.113.61:33669)
0619 06:57:48.797965 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.797969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.811583 (+ 13614us) server_negotiation.cc:407] Connection header received
0619 06:57:48.811827 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.811831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.811898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.811959 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.813105 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.813819 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.814980 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.815348 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.815435 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.817512 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.817517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.817520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.817550 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.819410 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.820017 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.820021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.820022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.820101 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.820429 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.820432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.820434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.820688 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.820831 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.821204 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.821344 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:57:48.826591 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be873e30984b42c987503cabb8a711ca. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.828125 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.828327 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be873e30984b42c987503cabb8a711ca. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.829401 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.831954 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.819681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.22.113.60:38793)
0619 06:57:48.819800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.819806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.822734 (+ 2928us) server_negotiation.cc:407] Connection header received
0619 06:57:48.822909 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.822912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.822954 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.823001 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.823885 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.824664 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.825963 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.826289 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.826358 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.828166 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.828170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.828173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.828197 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.829856 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.830473 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.830478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.830479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.830575 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.830884 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.830887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.830889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.831112 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.831295 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.831642 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.831796 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:57:48.832119 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:48.800687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.22.113.62:35419)
0619 06:57:48.801555 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.801560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.816188 (+ 14628us) server_negotiation.cc:407] Connection header received
0619 06:57:48.817523 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.817526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.817582 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.817634 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.818469 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.819193 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.821857 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:48.822157 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:48.822242 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:48.828398 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:48.828402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:48.828405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:48.828439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:48.829992 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.830635 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.830640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.830642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.830753 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:48.831073 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:48.831077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:48.831078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:48.831329 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:48.831450 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:48.831802 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 06:57:48.831935 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770}
W20260619 06:57:48.832276 13434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:57:48.832345 13434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:57:48.835201 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.835376 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:48.836546 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.836692 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.838018 13420 catalog_manager.cc:1347] Generated new cluster ID: edbc3286e48a4be7a959ee2f2a2cf8af
I20260619 06:57:48.838091 13420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:48.841261 13438 mvcc.cc:205] Tried to move back new op lower bound from 7298466893056602112 to 7298466892992909312. Current Snapshot: MvccSnapshot[applied={T|T < 7298466893056602112}]
I20260619 06:57:48.844650 13414 sys_catalog.cc:455] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.844792 13414 sys_catalog.cc:458] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:48.845891 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be873e30984b42c987503cabb8a711ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be873e30984b42c987503cabb8a711ca" member_type: VOTER last_known_addr { host: "127.22.113.62" port: 35419 } } peers { permanent_uuid: "2d0e12ac6ced48179537db91176babbb" member_type: VOTER last_known_addr { host: "127.22.113.61" port: 33669 } } peers { permanent_uuid: "a0e249cf962143e49f8870da973822c3" member_type: VOTER last_known_addr { host: "127.22.113.60" port: 38793 } } }
I20260619 06:57:48.846015 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:57:48.847103 13437 mvcc.cc:205] Tried to move back new op lower bound from 7298466893056602112 to 7298466892992909312. Current Snapshot: MvccSnapshot[applied={T|T < 7298466893056602112}]
I20260619 06:57:48.867652 13420 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:48.869036 13420 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:48.894979 13420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be873e30984b42c987503cabb8a711ca: Generated new TSK 0
I20260619 06:57:48.896256 13420 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:57:48.932559 13382 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.929158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.22.113.60:38793)
0619 06:57:48.929317 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.929322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.931347 (+ 2025us) server_negotiation.cc:407] Connection header received
0619 06:57:48.931565 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.931569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.931626 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.931683 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.932391 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40910: BlockingRecv error: recv got EOF from 127.0.0.1:40910 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:57:48.933960 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.930013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.22.113.62:35419)
0619 06:57:48.930141 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.930145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.933076 (+ 2931us) server_negotiation.cc:407] Connection header received
0619 06:57:48.933244 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.933247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.933298 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.933349 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.933829 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37662: BlockingRecv error: recv got EOF from 127.0.0.1:37662 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:57:48.935667 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.930723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.22.113.61:33669)
0619 06:57:48.930846 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.930851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.934625 (+ 3774us) server_negotiation.cc:407] Connection header received
0619 06:57:48.934860 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.934863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.934920 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.934976 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.935538 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45440: BlockingRecv error: recv got EOF from 127.0.0.1:45440 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:57:48.940340 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.938723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.22.113.61:33669)
0619 06:57:48.938830 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.938835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.939393 (+ 558us) server_negotiation.cc:407] Connection header received
0619 06:57:48.939615 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.939618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.939695 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.939757 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.940232 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45444: BlockingRecv error: recv got EOF from 127.0.0.1:45444 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 06:57:48.945569 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.944158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.22.113.61:33669)
0619 06:57:48.944258 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.944262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.944724 (+ 462us) server_negotiation.cc:407] Connection header received
0619 06:57:48.944907 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.944910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.944959 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.945010 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.945467 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45448: BlockingRecv error: recv got EOF from 127.0.0.1:45448 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 06:57:48.951383 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.949894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.22.113.61:33669)
0619 06:57:48.949987 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.949992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.950461 (+ 469us) server_negotiation.cc:407] Connection header received
0619 06:57:48.950639 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.950645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.950695 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.950759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.951286 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45454: BlockingRecv error: recv got EOF from 127.0.0.1:45454 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 06:57:48.959429 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.957904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.22.113.61:33669)
0619 06:57:48.958042 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.958049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.958512 (+ 463us) server_negotiation.cc:407] Connection header received
0619 06:57:48.958691 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.958697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.958761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.958816 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.959325 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45470: BlockingRecv error: recv got EOF from 127.0.0.1:45470 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 06:57:48.972559 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.970596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.22.113.61:33669)
0619 06:57:48.970745 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.970750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.971435 (+ 685us) server_negotiation.cc:407] Connection header received
0619 06:57:48.971661 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.971665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.971736 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.971796 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.972412 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45476: BlockingRecv error: recv got EOF from 127.0.0.1:45476 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:57:48.993781 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:48.992026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.22.113.61:33669)
0619 06:57:48.992209 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:48.992214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:48.992601 (+ 387us) server_negotiation.cc:407] Connection header received
0619 06:57:48.992835 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:48.992839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:48.992903 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:48.992960 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:48.993622 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45486: BlockingRecv error: recv got EOF from 127.0.0.1:45486 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 06:57:49.030301 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:49.028973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.22.113.61:33669)
0619 06:57:49.029154 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:49.029159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:49.029248 (+ 89us) server_negotiation.cc:407] Connection header received
0619 06:57:49.029391 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:49.029395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:49.029469 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:49.029621 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:49.030124 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45490: BlockingRecv error: recv got EOF from 127.0.0.1:45490 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 06:57:49.097990 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:49.096683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.22.113.61:33669)
0619 06:57:49.096936 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:49.096940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:49.097031 (+ 91us) server_negotiation.cc:407] Connection header received
0619 06:57:49.097171 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:49.097174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:49.097228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:49.097317 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:49.097855 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45496: BlockingRecv error: recv got EOF from 127.0.0.1:45496 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260619 06:57:49.229895 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:49.228499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.22.113.61:33669)
0619 06:57:49.228680 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:49.228686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:49.228869 (+ 183us) server_negotiation.cc:407] Connection header received
0619 06:57:49.229074 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:49.229077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:49.229131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:49.229235 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:49.229727 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45502: BlockingRecv error: recv got EOF from 127.0.0.1:45502 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 06:57:49.489975 13305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:57:49.488605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.22.113.61:33669)
0619 06:57:49.488757 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:49.488763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:49.488892 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:57:49.489093 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:49.489097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:49.489158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:49.489268 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:49.489815 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45514: BlockingRecv error: recv got EOF from 127.0.0.1:45514 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:57:49.783949 13411 catalog_manager.cc:1259] Loaded cluster ID: edbc3286e48a4be7a959ee2f2a2cf8af
I20260619 06:57:49.784036 13411 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: loading cluster ID for follower catalog manager: success
I20260619 06:57:49.785619 13411 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: acquiring CA information for follower catalog manager: success
I20260619 06:57:49.786716 13411 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d0e12ac6ced48179537db91176babbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:57:49.833956 13434 catalog_manager.cc:1259] Loaded cluster ID: edbc3286e48a4be7a959ee2f2a2cf8af
I20260619 06:57:49.834061 13434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: loading cluster ID for follower catalog manager: success
I20260619 06:57:49.836040 13434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: acquiring CA information for follower catalog manager: success
I20260619 06:57:49.837339 13434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0e249cf962143e49f8870da973822c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:57:50.008630 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:50.004519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.22.113.61:33669)
0619 06:57:50.004752 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:50.004755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:50.004928 (+ 173us) server_negotiation.cc:407] Connection header received
0619 06:57:50.005010 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:50.005012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:50.005144 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:50.005293 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:50.005574 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.006110 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.007044 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.007348 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.007434 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:50.007719 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:50.008154 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:50.008251 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:50.008447 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:57:50.008499 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260619 06:57:50.016177 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:50.012277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.22.113.60:38793)
0619 06:57:50.012573 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:50.012578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:50.012593 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:57:50.012777 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:50.012780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:50.012911 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:50.013051 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:50.013293 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.013818 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.014697 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.014934 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.015052 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:50.015361 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:50.015758 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:50.015900 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:50.015940 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 06:57:50.015994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260619 06:57:50.024246 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:50.020047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.22.113.60:38793)
0619 06:57:50.020168 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:50.020174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:50.020506 (+ 332us) server_negotiation.cc:407] Connection header received
0619 06:57:50.020680 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:50.020686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:50.020767 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:50.020840 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:50.021228 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.021755 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.022982 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.023342 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.023450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:50.023699 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:50.023802 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:50.023902 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:50.023992 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:57:50.024052 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:57:50.024562 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:50.020114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.22.113.61:33669)
0619 06:57:50.020258 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:50.020261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:50.020415 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:57:50.020615 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:50.020619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:50.020676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:50.020754 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:50.021155 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.022017 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.023145 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.023540 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.023658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:50.024054 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:50.024132 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:50.024177 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:50.024345 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 06:57:50.024431 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:57:50.025015 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:50.019809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.22.113.62:35419)
0619 06:57:50.020260 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:50.020264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:50.020363 (+ 99us) server_negotiation.cc:407] Connection header received
0619 06:57:50.020621 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:50.020624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:50.020781 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:50.020908 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:57:50.021301 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.022117 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.023041 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:50.023464 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:50.023556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:50.023918 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:57:50.024291 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:57:50.024389 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:50.024769 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 06:57:50.024845 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":85,"threads_started":1}
I20260619 06:57:50.033785 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 13158
I20260619 06:57:50.040788 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 13233
I20260619 06:57:50.046955 22980 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9tRik3/build/debug/bin/kudu with pid 13308
2026-06-19T06:57:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3174 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6419 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293516 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST